[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786332336319544-28292-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 03:25:36 dist-test-slave-vpvm krb5kdc[28300](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 03:25:36 dist-test-slave-vpvm krb5kdc[28300](info): set up 2 sockets
Aug 10 03:25:36 dist-test-slave-vpvm krb5kdc[28300](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260810 03:25:37.928124 28292 mini_kdc.cc:121] Time spent starting KDC: real 1.429s user 0.003s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786332336319544-28292-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786332336319544-28292-0/krb5kdc/test-user.keytab.
Aug 10 03:25:38 dist-test-slave-vpvm krb5kdc[28300](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332338, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T03:25:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T03:25:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.161.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.161.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.161.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.62.keytab.
Entry for principal kudu/127.27.161.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.62.keytab.
I20260810 03:25:38.144436 28292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iBH5/build/asan/bin/kudu
/tmp/dist-test-task24iBH5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786332336319544-28292-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786332336319544-28292-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786332336319544-28292-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786332336319544-28292-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.161.62:41293
--webserver_interface=127.27.161.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.62.keytab
--principal=kudu/127.27.161.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.161.20:38245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.161.62:41293
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786332336319544-28292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786332336319544-28292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786332336319544-28292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:25:38.641465 28321 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:25:38.641920 28321 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:25:38.642081 28321 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:25:38.651001 28321 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:25:38.651132 28321 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:25:38.651194 28321 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:25:38.651259 28321 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:25:38.651360 28321 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:25:38.663688 28321 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.161.20:38245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786332336319544-28292-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786332336319544-28292-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.161.62:41293
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.62.keytab
--principal=kudu/127.27.161.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.161.62:41293
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786332336319544-28292-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.161.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786332336319544-28292-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:25:38.665717 28321 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:25:38.667789 28321 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:25:38.690829 28330 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:25:38.690943 28328 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:25:38.691331 28327 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:25:38.691967 28321 server_base.cc:1034] running on GCE node
I20260810 03:25:38.693140 28321 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:25:38.695854 28321 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:25:38.697216 28321 hybrid_clock.cc:648] HybridClock initialized: now 1786332338697096 us; error 148 us; skew 500 ppm
I20260810 03:25:38.697849 28321 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:25:38 dist-test-slave-vpvm krb5kdc[28300](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332338, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.161.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:25:38.705528 28321 init.cc:377] Logged in from keytab as kudu/127.27.161.62@KRBTEST.COM (short username kudu)
I20260810 03:25:38.708487 28321 webserver.cc:460] Webserver started at http://127.27.161.62:35093/ using document root <none> and password file <none>
I20260810 03:25:38.709719 28321 fs_manager.cc:359] Metadata directory not provided
I20260810 03:25:38.709865 28321 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:25:38.710413 28321 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:25:38.715385 28321 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786332336319544-28292-0/minicluster-data/master-0/data/instance:
uuid: "6276ade8ce0a4f05b8d05baf7b80624e"
format_stamp: "Formatted at 2026-08-10 03:25:38 on dist-test-slave-vpvm"
server_key: "a3981a723fe86d564e65814eda066ce4"
server_key_iv: "acaff16e8b34a63a17a7e0ee7ac53f05"
server_key_version: "encryptionkey@0"
I20260810 03:25:38.716719 28321 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786332336319544-28292-0/minicluster-data/master-0/wal/instance:
uuid: "6276ade8ce0a4f05b8d05baf7b80624e"
format_stamp: "Formatted at 2026-08-10 03:25:38 on dist-test-slave-vpvm"
server_key: "a3981a723fe86d564e65814eda066ce4"
server_key_iv: "acaff16e8b34a63a17a7e0ee7ac53f05"
server_key_version: "encryptionkey@0"
I20260810 03:25:38.725271 28321 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.005s sys 0.005s
I20260810 03:25:38.731480 28340 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:25:38.733590 28321 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.001s
I20260810 03:25:38.733880 28321 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786332336319544-28292-0/minicluster-data/master-0/data,/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786332336319544-28292-0/minicluster-data/master-0/wal
uuid: "6276ade8ce0a4f05b8d05baf7b80624e"
format_stamp: "Formatted at 2026-08-10 03:25:38 on dist-test-slave-vpvm"
server_key: "a3981a723fe86d564e65814eda066ce4"
server_key_iv: "acaff16e8b34a63a17a7e0ee7ac53f05"
server_key_version: "encryptionkey@0"
I20260810 03:25:38.734165 28321 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786332336319544-28292-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786332336319544-28292-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786332336319544-28292-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:25:38.754838 28321 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:25:38.761044 28321 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:25:38.761523 28321 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:25:38.793222 28321 rpc_server.cc:307] RPC server started. Bound to: 127.27.161.62:41293
I20260810 03:25:38.793372 28413 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.161.62:41293 every 8 connection(s)
I20260810 03:25:38.795997 28321 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786332336319544-28292-0/minicluster-data/master-0/data/info.pb
I20260810 03:25:38.799460 28292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iBH5/build/asan/bin/kudu as pid 28321
I20260810 03:25:38.799892 28292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786332336319544-28292-0/minicluster-data/master-0/wal/instance
I20260810 03:25:38.802152 28292 external_mini_cluster.cc:1428] Setting key 89b2305815c2477c644fab64f02c46ce
I20260810 03:25:38.803560 28415 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:25:38.821339 28415 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6276ade8ce0a4f05b8d05baf7b80624e: Bootstrap starting.
I20260810 03:25:38.827723 28415 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6276ade8ce0a4f05b8d05baf7b80624e: Neither blocks nor log segments found. Creating new log.
I20260810 03:25:38.829711 28415 log.cc:824] T 00000000000000000000000000000000 P 6276ade8ce0a4f05b8d05baf7b80624e: Log is configured to *not* fsync() on all Append() calls
I20260810 03:25:38.835191 28415 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6276ade8ce0a4f05b8d05baf7b80624e: No bootstrap required, opened a new log
Aug 10 03:25:38 dist-test-slave-vpvm krb5kdc[28300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332338, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.161.62@KRBTEST.COM
I20260810 03:25:38.847064 28415 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6276ade8ce0a4f05b8d05baf7b80624e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6276ade8ce0a4f05b8d05baf7b80624e" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 41293 } }
I20260810 03:25:38.847899 28415 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6276ade8ce0a4f05b8d05baf7b80624e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:25:38.848125 28415 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6276ade8ce0a4f05b8d05baf7b80624e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6276ade8ce0a4f05b8d05baf7b80624e, State: Initialized, Role: FOLLOWER
I20260810 03:25:38.849730 28415 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6276ade8ce0a4f05b8d05baf7b80624e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6276ade8ce0a4f05b8d05baf7b80624e" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 41293 } }
I20260810 03:25:38.850342 28415 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6276ade8ce0a4f05b8d05baf7b80624e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 03:25:38.850694 28415 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6276ade8ce0a4f05b8d05baf7b80624e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 03:25:38.851055 28415 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6276ade8ce0a4f05b8d05baf7b80624e [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:25:38.852351 28417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:38.807968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33452 (local address 127.27.161.62:41293)
0810 03:25:38.809185 (+ 1217us) server_negotiation.cc:206] Beginning negotiation
0810 03:25:38.809202 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0810 03:25:38.810570 (+ 1368us) server_negotiation.cc:407] Connection header received
0810 03:25:38.821944 (+ 11374us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:25:38.822021 (+ 77us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:25:38.822637 (+ 616us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:25:38.823093 (+ 456us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:25:38.827038 (+ 3945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:38.829102 (+ 2064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:38.831996 (+ 2894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:38.833236 (+ 1240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:38.833536 (+ 300us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:25:38.840946 (+ 7410us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:25:38.841011 (+ 65us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:25:38.841050 (+ 39us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:25:38.841101 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:25:38.845315 (+ 4214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:25:38.846521 (+ 1206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:25:38.846529 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:25:38.846535 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:25:38.846731 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:25:38.847270 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:25:38.847274 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:25:38.847276 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:25:38.848365 (+ 1089us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:25:38.848790 (+ 425us) server_negotiation.cc:1035] Waiting for connection context
0810 03:25:38.850217 (+ 1427us) server_negotiation.cc:299] Negotiation successful
0810 03:25:38.850622 (+ 405us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":855,"thread_start_us":429,"threads_started":1}
I20260810 03:25:38.855067 28415 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6276ade8ce0a4f05b8d05baf7b80624e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6276ade8ce0a4f05b8d05baf7b80624e" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 41293 } }
I20260810 03:25:38.856573 28415 leader_election.cc:302] T 00000000000000000000000000000000 P 6276ade8ce0a4f05b8d05baf7b80624e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6276ade8ce0a4f05b8d05baf7b80624e; no voters:
I20260810 03:25:38.857780 28415 leader_election.cc:288] T 00000000000000000000000000000000 P 6276ade8ce0a4f05b8d05baf7b80624e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 03:25:38.858126 28423 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6276ade8ce0a4f05b8d05baf7b80624e [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:25:38.859594 28423 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6276ade8ce0a4f05b8d05baf7b80624e [term 1 LEADER]: Becoming Leader. State: Replica: 6276ade8ce0a4f05b8d05baf7b80624e, State: Running, Role: LEADER
I20260810 03:25:38.860422 28423 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6276ade8ce0a4f05b8d05baf7b80624e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6276ade8ce0a4f05b8d05baf7b80624e" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 41293 } }
I20260810 03:25:38.861320 28415 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6276ade8ce0a4f05b8d05baf7b80624e [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:25:38.866966 28427 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6276ade8ce0a4f05b8d05baf7b80624e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6276ade8ce0a4f05b8d05baf7b80624e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6276ade8ce0a4f05b8d05baf7b80624e" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 41293 } } }
I20260810 03:25:38.867419 28427 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6276ade8ce0a4f05b8d05baf7b80624e [sys.catalog]: This master's current role is: LEADER
I20260810 03:25:38.868074 28429 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6276ade8ce0a4f05b8d05baf7b80624e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6276ade8ce0a4f05b8d05baf7b80624e. Latest consensus state: current_term: 1 leader_uuid: "6276ade8ce0a4f05b8d05baf7b80624e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6276ade8ce0a4f05b8d05baf7b80624e" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 41293 } } }
I20260810 03:25:38.868438 28429 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6276ade8ce0a4f05b8d05baf7b80624e [sys.catalog]: This master's current role is: LEADER
I20260810 03:25:38.871230 28436 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 03:25:38.880712 28436 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 03:25:38.897501 28436 catalog_manager.cc:1347] Generated new cluster ID: fad080f996b449a2b6c2d40f74168537
I20260810 03:25:38.897716 28436 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 03:25:38.931135 28436 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 03:25:38.933459 28436 catalog_manager.cc:1504] Loading token signing keys...
I20260810 03:25:38.951633 28436 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6276ade8ce0a4f05b8d05baf7b80624e: Generated new TSK 0
I20260810 03:25:38.953393 28436 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.161.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.161.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.161.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.1.keytab.
Entry for principal kudu/127.27.161.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.1.keytab.
I20260810 03:25:39.021525 28292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iBH5/build/asan/bin/kudu
/tmp/dist-test-task24iBH5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786332336319544-28292-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786332336319544-28292-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786332336319544-28292-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786332336319544-28292-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.161.1:0
--local_ip_for_outbound_sockets=127.27.161.1
--webserver_interface=127.27.161.1
--webserver_port=0
--tserver_master_addrs=127.27.161.62:41293
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.1.keytab
--principal=kudu/127.27.161.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.161.20:38245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786332336319544-28292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786332336319544-28292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786332336319544-28292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:25:39.474392 28458 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:25:39.474715 28458 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:25:39.474908 28458 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:25:39.482975 28458 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:25:39.483119 28458 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:25:39.483300 28458 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.161.1
I20260810 03:25:39.494424 28458 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.161.20:38245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786332336319544-28292-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786332336319544-28292-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.1.keytab
--principal=kudu/127.27.161.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.161.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786332336319544-28292-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.161.1
--webserver_port=0
--tserver_master_addrs=127.27.161.62:41293
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.161.1
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786332336319544-28292-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:25:39.495844 28458 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:25:39.497437 28458 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:25:39.512180 28465 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:25:39.512133 28467 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:25:39.512099 28469 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:25:39.512974 28458 server_base.cc:1034] running on GCE node
I20260810 03:25:39.513679 28458 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:25:39.514724 28458 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:25:39.515919 28458 hybrid_clock.cc:648] HybridClock initialized: now 1786332339515856 us; error 59 us; skew 500 ppm
I20260810 03:25:39.516235 28458 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:25:39 dist-test-slave-vpvm krb5kdc[28300](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332339, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.161.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:25:39.519591 28458 init.cc:377] Logged in from keytab as kudu/127.27.161.1@KRBTEST.COM (short username kudu)
I20260810 03:25:39.521260 28458 webserver.cc:460] Webserver started at http://127.27.161.1:46005/ using document root <none> and password file <none>
I20260810 03:25:39.522044 28458 fs_manager.cc:359] Metadata directory not provided
I20260810 03:25:39.522181 28458 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:25:39.522500 28458 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:25:39.525851 28458 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786332336319544-28292-0/minicluster-data/ts-0/data/instance:
uuid: "ef2d467290524564bab2d0141217702a"
format_stamp: "Formatted at 2026-08-10 03:25:39 on dist-test-slave-vpvm"
server_key: "e332c21fa428aacee792298b31ddb134"
server_key_iv: "fb839ca4398c194f230535fe513e0960"
server_key_version: "encryptionkey@0"
I20260810 03:25:39.526808 28458 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786332336319544-28292-0/minicluster-data/ts-0/wal/instance:
uuid: "ef2d467290524564bab2d0141217702a"
format_stamp: "Formatted at 2026-08-10 03:25:39 on dist-test-slave-vpvm"
server_key: "e332c21fa428aacee792298b31ddb134"
server_key_iv: "fb839ca4398c194f230535fe513e0960"
server_key_version: "encryptionkey@0"
I20260810 03:25:39.533020 28458 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260810 03:25:39.537259 28478 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:25:39.538877 28458 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 03:25:39.539069 28458 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786332336319544-28292-0/minicluster-data/ts-0/data,/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786332336319544-28292-0/minicluster-data/ts-0/wal
uuid: "ef2d467290524564bab2d0141217702a"
format_stamp: "Formatted at 2026-08-10 03:25:39 on dist-test-slave-vpvm"
server_key: "e332c21fa428aacee792298b31ddb134"
server_key_iv: "fb839ca4398c194f230535fe513e0960"
server_key_version: "encryptionkey@0"
I20260810 03:25:39.539309 28458 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786332336319544-28292-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786332336319544-28292-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786332336319544-28292-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:25:39.559276 28458 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:25:39.563757 28458 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:25:39.564045 28458 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:25:39.565359 28458 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:25:39.567298 28458 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:25:39.567389 28458 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:25:39.567546 28458 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:25:39.567608 28458 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:25:39.617863 28458 rpc_server.cc:307] RPC server started. Bound to: 127.27.161.1:33403
I20260810 03:25:39.618018 28646 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.161.1:33403 every 8 connection(s)
I20260810 03:25:39.619258 28458 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786332336319544-28292-0/minicluster-data/ts-0/data/info.pb
I20260810 03:25:39.627219 28292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iBH5/build/asan/bin/kudu as pid 28458
I20260810 03:25:39.627533 28292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786332336319544-28292-0/minicluster-data/ts-0/wal/instance
I20260810 03:25:39.628225 28292 external_mini_cluster.cc:1428] Setting key c918e8358e0280e4cdb803a11bf79b1e
Aug 10 03:25:39 dist-test-slave-vpvm krb5kdc[28300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332339, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.161.1@KRBTEST.COM for kudu/127.27.161.62@KRBTEST.COM
I20260810 03:25:39.646871 28656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:39.623257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.161.1:47541 (local address 127.27.161.62:41293)
0810 03:25:39.623777 (+ 520us) server_negotiation.cc:206] Beginning negotiation
0810 03:25:39.623781 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:25:39.625086 (+ 1305us) server_negotiation.cc:407] Connection header received
0810 03:25:39.627429 (+ 2343us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:25:39.627435 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:25:39.627524 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:25:39.627623 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:25:39.629516 (+ 1893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:39.630678 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:39.633343 (+ 2665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:39.633888 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:39.634039 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:25:39.639131 (+ 5092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:25:39.639139 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:25:39.639145 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:25:39.639194 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:25:39.643207 (+ 4013us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:25:39.644122 (+ 915us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:25:39.644127 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:25:39.644130 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:25:39.644286 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:25:39.644759 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:25:39.644764 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:25:39.644767 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:25:39.645230 (+ 463us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:25:39.645400 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0810 03:25:39.646216 (+ 816us) server_negotiation.cc:299] Negotiation successful
0810 03:25:39.646493 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":408,"thread_start_us":301,"threads_started":1}
I20260810 03:25:39.647961 28657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:39.623832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:41293 (local address 127.27.161.1:47541)
0810 03:25:39.624899 (+ 1067us) negotiation.cc:107] Waiting for socket to connect
0810 03:25:39.624952 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0810 03:25:39.627136 (+ 2184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:25:39.627907 (+ 771us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:25:39.627915 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:25:39.628505 (+ 590us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:25:39.629241 (+ 736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:25:39.629269 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:25:39.630854 (+ 1585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:25:39.630860 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:25:39.633166 (+ 2306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:25:39.633184 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:25:39.634198 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:25:39.634207 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:25:39.634560 (+ 353us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:25:39.635545 (+ 985us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:25:39.635595 (+ 50us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:25:39.638827 (+ 3232us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:25:39.643398 (+ 4571us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:25:39.643406 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:25:39.643421 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:25:39.643994 (+ 573us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:25:39.644404 (+ 410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:25:39.644408 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:25:39.644410 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:25:39.644636 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:25:39.645342 (+ 706us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:25:39.645348 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:25:39.645766 (+ 418us) client_negotiation.cc:769] Sending connection context
0810 03:25:39.646099 (+ 333us) client_negotiation.cc:240] Negotiation successful
0810 03:25:39.646581 (+ 482us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":766,"thread_start_us":439,"threads_started":1}
I20260810 03:25:39.649793 28650 heartbeater.cc:347] Connected to a master server at 127.27.161.62:41293
I20260810 03:25:39.650322 28650 heartbeater.cc:464] Registering TS with master...
I20260810 03:25:39.651511 28650 heartbeater.cc:511] Master 127.27.161.62:41293 requested a full tablet report, sending...
I20260810 03:25:39.656133 28361 ts_manager.cc:194] Registered new tserver with Master: ef2d467290524564bab2d0141217702a (127.27.161.1:33403)
I20260810 03:25:39.659286 28361 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.161.1@KRBTEST.COM'} at 127.27.161.1:47541
WARNING: no policy specified for kudu/127.27.161.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.161.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.161.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.2.keytab.
Entry for principal kudu/127.27.161.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.2.keytab.
I20260810 03:25:39.697213 28292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iBH5/build/asan/bin/kudu
/tmp/dist-test-task24iBH5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786332336319544-28292-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786332336319544-28292-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786332336319544-28292-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786332336319544-28292-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.161.2:0
--local_ip_for_outbound_sockets=127.27.161.2
--webserver_interface=127.27.161.2
--webserver_port=0
--tserver_master_addrs=127.27.161.62:41293
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.2.keytab
--principal=kudu/127.27.161.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.161.20:38245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786332336319544-28292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786332336319544-28292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786332336319544-28292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:25:40.126523 28660 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:25:40.126819 28660 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:25:40.127002 28660 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:25:40.135324 28660 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:25:40.135499 28660 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:25:40.135687 28660 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.161.2
I20260810 03:25:40.147087 28660 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.161.20:38245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786332336319544-28292-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786332336319544-28292-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.2.keytab
--principal=kudu/127.27.161.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.161.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786332336319544-28292-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.161.2
--webserver_port=0
--tserver_master_addrs=127.27.161.62:41293
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.161.2
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786332336319544-28292-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:25:40.148512 28660 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:25:40.150225 28660 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:25:40.165825 28665 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:25:40.165792 28667 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:25:40.165735 28669 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:25:40.167183 28660 server_base.cc:1034] running on GCE node
I20260810 03:25:40.167860 28660 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:25:40.169018 28660 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:25:40.170272 28660 hybrid_clock.cc:648] HybridClock initialized: now 1786332340170227 us; error 68 us; skew 500 ppm
I20260810 03:25:40.170656 28660 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:25:40 dist-test-slave-vpvm krb5kdc[28300](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332340, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.161.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:25:40.174358 28660 init.cc:377] Logged in from keytab as kudu/127.27.161.2@KRBTEST.COM (short username kudu)
I20260810 03:25:40.176375 28660 webserver.cc:460] Webserver started at http://127.27.161.2:45405/ using document root <none> and password file <none>
I20260810 03:25:40.177233 28660 fs_manager.cc:359] Metadata directory not provided
I20260810 03:25:40.177376 28660 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:25:40.177716 28660 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:25:40.181231 28660 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786332336319544-28292-0/minicluster-data/ts-1/data/instance:
uuid: "35057bc5f8af45a2a0f0f3fa9af618ed"
format_stamp: "Formatted at 2026-08-10 03:25:40 on dist-test-slave-vpvm"
server_key: "c6115e5683928fe937efb005b01e9ac8"
server_key_iv: "8112aeb37ffabe2293bdeab783be2caf"
server_key_version: "encryptionkey@0"
I20260810 03:25:40.182166 28660 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786332336319544-28292-0/minicluster-data/ts-1/wal/instance:
uuid: "35057bc5f8af45a2a0f0f3fa9af618ed"
format_stamp: "Formatted at 2026-08-10 03:25:40 on dist-test-slave-vpvm"
server_key: "c6115e5683928fe937efb005b01e9ac8"
server_key_iv: "8112aeb37ffabe2293bdeab783be2caf"
server_key_version: "encryptionkey@0"
I20260810 03:25:40.188498 28660 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.004s
I20260810 03:25:40.192924 28679 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:25:40.194747 28660 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260810 03:25:40.194949 28660 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786332336319544-28292-0/minicluster-data/ts-1/data,/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786332336319544-28292-0/minicluster-data/ts-1/wal
uuid: "35057bc5f8af45a2a0f0f3fa9af618ed"
format_stamp: "Formatted at 2026-08-10 03:25:40 on dist-test-slave-vpvm"
server_key: "c6115e5683928fe937efb005b01e9ac8"
server_key_iv: "8112aeb37ffabe2293bdeab783be2caf"
server_key_version: "encryptionkey@0"
I20260810 03:25:40.195184 28660 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786332336319544-28292-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786332336319544-28292-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786332336319544-28292-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:25:40.217942 28660 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:25:40.223067 28660 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:25:40.223444 28660 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:25:40.224757 28660 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:25:40.226454 28660 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:25:40.226548 28660 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:25:40.226680 28660 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:25:40.226745 28660 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:25:40.276073 28660 rpc_server.cc:307] RPC server started. Bound to: 127.27.161.2:44723
I20260810 03:25:40.276160 28836 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.161.2:44723 every 8 connection(s)
I20260810 03:25:40.277518 28660 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786332336319544-28292-0/minicluster-data/ts-1/data/info.pb
I20260810 03:25:40.284937 28292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iBH5/build/asan/bin/kudu as pid 28660
I20260810 03:25:40.285209 28292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786332336319544-28292-0/minicluster-data/ts-1/wal/instance
I20260810 03:25:40.285765 28292 external_mini_cluster.cc:1428] Setting key ec3b747ca9b8a5c31dc59a2f9a34b0e2
Aug 10 03:25:40 dist-test-slave-vpvm krb5kdc[28300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332340, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.161.2@KRBTEST.COM for kudu/127.27.161.62@KRBTEST.COM
I20260810 03:25:40.302676 28839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:40.280643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.161.2:48409 (local address 127.27.161.62:41293)
0810 03:25:40.281101 (+ 458us) server_negotiation.cc:206] Beginning negotiation
0810 03:25:40.281106 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:25:40.282159 (+ 1053us) server_negotiation.cc:407] Connection header received
0810 03:25:40.284473 (+ 2314us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:25:40.284479 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:25:40.284569 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:25:40.284676 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:25:40.286548 (+ 1872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:40.287566 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:40.289369 (+ 1803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:40.290055 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:40.290182 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:25:40.295107 (+ 4925us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:25:40.295113 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:25:40.295117 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:25:40.295159 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:25:40.298679 (+ 3520us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:25:40.299628 (+ 949us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:25:40.299634 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:25:40.299637 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:25:40.299888 (+ 251us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:25:40.300407 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:25:40.300414 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:25:40.300417 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:25:40.300914 (+ 497us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:25:40.301083 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0810 03:25:40.301973 (+ 890us) server_negotiation.cc:299] Negotiation successful
0810 03:25:40.302274 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":354,"thread_start_us":240,"threads_started":1}
I20260810 03:25:40.303674 28841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:40.281101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:41293 (local address 127.27.161.2:48409)
0810 03:25:40.281990 (+ 889us) negotiation.cc:107] Waiting for socket to connect
0810 03:25:40.282041 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0810 03:25:40.284114 (+ 2073us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:25:40.284945 (+ 831us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:25:40.284954 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:25:40.285559 (+ 605us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:25:40.286351 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:25:40.286373 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:25:40.287720 (+ 1347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:25:40.287724 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:25:40.289216 (+ 1492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:25:40.289230 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:25:40.290592 (+ 1362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:25:40.290602 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:25:40.290926 (+ 324us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:25:40.291765 (+ 839us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:25:40.291802 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:25:40.294806 (+ 3004us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:25:40.298886 (+ 4080us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:25:40.298898 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:25:40.298920 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:25:40.299452 (+ 532us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:25:40.300037 (+ 585us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:25:40.300041 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:25:40.300043 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:25:40.300268 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:25:40.301052 (+ 784us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:25:40.301058 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:25:40.301432 (+ 374us) client_negotiation.cc:769] Sending connection context
0810 03:25:40.301754 (+ 322us) client_negotiation.cc:240] Negotiation successful
0810 03:25:40.302288 (+ 534us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":611,"thread_start_us":374,"threads_started":1}
I20260810 03:25:40.305310 28837 heartbeater.cc:347] Connected to a master server at 127.27.161.62:41293
I20260810 03:25:40.305627 28837 heartbeater.cc:464] Registering TS with master...
I20260810 03:25:40.306689 28837 heartbeater.cc:511] Master 127.27.161.62:41293 requested a full tablet report, sending...
I20260810 03:25:40.308465 28361 ts_manager.cc:194] Registered new tserver with Master: 35057bc5f8af45a2a0f0f3fa9af618ed (127.27.161.2:44723)
I20260810 03:25:40.309396 28361 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.161.2@KRBTEST.COM'} at 127.27.161.2:48409
WARNING: no policy specified for kudu/127.27.161.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.161.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.161.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.3.keytab.
Entry for principal kudu/127.27.161.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.3.keytab.
I20260810 03:25:40.350338 28292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iBH5/build/asan/bin/kudu
/tmp/dist-test-task24iBH5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786332336319544-28292-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786332336319544-28292-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786332336319544-28292-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786332336319544-28292-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.161.3:0
--local_ip_for_outbound_sockets=127.27.161.3
--webserver_interface=127.27.161.3
--webserver_port=0
--tserver_master_addrs=127.27.161.62:41293
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.3.keytab
--principal=kudu/127.27.161.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.161.20:38245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786332336319544-28292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786332336319544-28292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786332336319544-28292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 03:25:40.664609 28650 heartbeater.cc:503] Master 127.27.161.62:41293 was elected leader, sending a full tablet report...
W20260810 03:25:40.735710 28844 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:25:40.736061 28844 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:25:40.736229 28844 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:25:40.744143 28844 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:25:40.744295 28844 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:25:40.744477 28844 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.161.3
I20260810 03:25:40.755314 28844 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.161.20:38245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786332336319544-28292-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786332336319544-28292-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.3.keytab
--principal=kudu/127.27.161.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.161.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786332336319544-28292-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.161.3
--webserver_port=0
--tserver_master_addrs=127.27.161.62:41293
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.161.3
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786332336319544-28292-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:25:40.756875 28844 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:25:40.758581 28844 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:25:40.774633 28855 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:25:40.774662 28858 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:25:40.775442 28854 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:25:40.775879 28844 server_base.cc:1034] running on GCE node
I20260810 03:25:40.776578 28844 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:25:40.777668 28844 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:25:40.778888 28844 hybrid_clock.cc:648] HybridClock initialized: now 1786332340778829 us; error 46 us; skew 500 ppm
I20260810 03:25:40.779245 28844 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:25:40 dist-test-slave-vpvm krb5kdc[28300](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332340, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.161.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:25:40.782706 28844 init.cc:377] Logged in from keytab as kudu/127.27.161.3@KRBTEST.COM (short username kudu)
I20260810 03:25:40.784449 28844 webserver.cc:460] Webserver started at http://127.27.161.3:34057/ using document root <none> and password file <none>
I20260810 03:25:40.785200 28844 fs_manager.cc:359] Metadata directory not provided
I20260810 03:25:40.785329 28844 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:25:40.785665 28844 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:25:40.789064 28844 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786332336319544-28292-0/minicluster-data/ts-2/data/instance:
uuid: "c8f1b3f243544f4993a276d2647fda0d"
format_stamp: "Formatted at 2026-08-10 03:25:40 on dist-test-slave-vpvm"
server_key: "37fba1591cc9e8d3109d1a274a563790"
server_key_iv: "3a8620fe164765567690982e02000243"
server_key_version: "encryptionkey@0"
I20260810 03:25:40.790073 28844 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786332336319544-28292-0/minicluster-data/ts-2/wal/instance:
uuid: "c8f1b3f243544f4993a276d2647fda0d"
format_stamp: "Formatted at 2026-08-10 03:25:40 on dist-test-slave-vpvm"
server_key: "37fba1591cc9e8d3109d1a274a563790"
server_key_iv: "3a8620fe164765567690982e02000243"
server_key_version: "encryptionkey@0"
I20260810 03:25:40.796685 28844 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260810 03:25:40.801059 28866 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:25:40.802826 28844 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 03:25:40.803030 28844 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786332336319544-28292-0/minicluster-data/ts-2/data,/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786332336319544-28292-0/minicluster-data/ts-2/wal
uuid: "c8f1b3f243544f4993a276d2647fda0d"
format_stamp: "Formatted at 2026-08-10 03:25:40 on dist-test-slave-vpvm"
server_key: "37fba1591cc9e8d3109d1a274a563790"
server_key_iv: "3a8620fe164765567690982e02000243"
server_key_version: "encryptionkey@0"
I20260810 03:25:40.803350 28844 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786332336319544-28292-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786332336319544-28292-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786332336319544-28292-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:25:40.829084 28844 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:25:40.833909 28844 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:25:40.834321 28844 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:25:40.835424 28844 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:25:40.837081 28844 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:25:40.837162 28844 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:25:40.837267 28844 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:25:40.837312 28844 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:25:40.885820 28844 rpc_server.cc:307] RPC server started. Bound to: 127.27.161.3:37873
I20260810 03:25:40.885871 29021 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.161.3:37873 every 8 connection(s)
I20260810 03:25:40.887308 28844 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786332336319544-28292-0/minicluster-data/ts-2/data/info.pb
I20260810 03:25:40.895787 28292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iBH5/build/asan/bin/kudu as pid 28844
I20260810 03:25:40.896023 28292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786332336319544-28292-0/minicluster-data/ts-2/wal/instance
I20260810 03:25:40.896497 28292 external_mini_cluster.cc:1428] Setting key 1dd18b7336e3c2f93ab7300d607c1dba
Aug 10 03:25:40 dist-test-slave-vpvm krb5kdc[28300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332340, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.161.3@KRBTEST.COM for kudu/127.27.161.62@KRBTEST.COM
I20260810 03:25:40.910786 29026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:40.890505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.161.3:47265 (local address 127.27.161.62:41293)
0810 03:25:40.891262 (+ 757us) server_negotiation.cc:206] Beginning negotiation
0810 03:25:40.891266 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:25:40.892862 (+ 1596us) server_negotiation.cc:407] Connection header received
0810 03:25:40.895363 (+ 2501us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:25:40.895370 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:25:40.895520 (+ 150us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:25:40.895829 (+ 309us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:25:40.897838 (+ 2009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:40.898837 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:40.900729 (+ 1892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:40.901189 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:40.901318 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:25:40.904747 (+ 3429us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:25:40.904753 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:25:40.904756 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:25:40.904792 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:25:40.907531 (+ 2739us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:25:40.908315 (+ 784us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:25:40.908320 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:25:40.908322 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:25:40.908472 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:25:40.908929 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:25:40.908935 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:25:40.908938 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:25:40.909247 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:25:40.909384 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0810 03:25:40.910131 (+ 747us) server_negotiation.cc:299] Negotiation successful
0810 03:25:40.910391 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":635,"thread_start_us":525,"threads_started":1}
I20260810 03:25:40.911978 29032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:40.891239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:41293 (local address 127.27.161.3:47265)
0810 03:25:40.892338 (+ 1099us) negotiation.cc:107] Waiting for socket to connect
0810 03:25:40.892388 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0810 03:25:40.894962 (+ 2574us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:25:40.896163 (+ 1201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:25:40.896173 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:25:40.896813 (+ 640us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:25:40.897644 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:25:40.897674 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:25:40.898998 (+ 1324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:25:40.899003 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:25:40.900583 (+ 1580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:25:40.900601 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:25:40.901424 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:25:40.901429 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:25:40.901647 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:25:40.902287 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:25:40.902311 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:25:40.904512 (+ 2201us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:25:40.907679 (+ 3167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:25:40.907690 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:25:40.907710 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:25:40.908194 (+ 484us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:25:40.908600 (+ 406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:25:40.908606 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:25:40.908610 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:25:40.908817 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:25:40.909385 (+ 568us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:25:40.909393 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:25:40.909754 (+ 361us) client_negotiation.cc:769] Sending connection context
0810 03:25:40.910033 (+ 279us) client_negotiation.cc:240] Negotiation successful
0810 03:25:40.910628 (+ 595us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":745,"thread_start_us":518,"threads_started":1}
I20260810 03:25:40.913676 29022 heartbeater.cc:347] Connected to a master server at 127.27.161.62:41293
I20260810 03:25:40.914075 29022 heartbeater.cc:464] Registering TS with master...
I20260810 03:25:40.914939 29022 heartbeater.cc:511] Master 127.27.161.62:41293 requested a full tablet report, sending...
I20260810 03:25:40.916409 28361 ts_manager.cc:194] Registered new tserver with Master: c8f1b3f243544f4993a276d2647fda0d (127.27.161.3:37873)
I20260810 03:25:40.917274 28361 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.161.3@KRBTEST.COM'} at 127.27.161.3:47265
I20260810 03:25:40.926565 28292 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 10 03:25:40 dist-test-slave-vpvm krb5kdc[28300](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332340, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 10 03:25:40 dist-test-slave-vpvm krb5kdc[28300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332340, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.27.161.1@KRBTEST.COM
I20260810 03:25:40.995051 29037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:40.977474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46392 (local address 127.27.161.1:33403)
0810 03:25:40.978418 (+ 944us) server_negotiation.cc:206] Beginning negotiation
0810 03:25:40.978428 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 03:25:40.978456 (+ 28us) server_negotiation.cc:407] Connection header received
0810 03:25:40.978520 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:25:40.978527 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:25:40.978755 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:25:40.979081 (+ 326us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:25:40.981084 (+ 2003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:40.982229 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:40.983616 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:40.984294 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:40.984425 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:25:40.988974 (+ 4549us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:25:40.988991 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:25:40.989009 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:25:40.989055 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:25:40.992042 (+ 2987us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:25:40.992808 (+ 766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:25:40.992816 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:25:40.992818 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:25:40.992959 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:25:40.993375 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:25:40.993380 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:25:40.993382 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:25:40.993883 (+ 501us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:25:40.994211 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0810 03:25:40.994499 (+ 288us) server_negotiation.cc:299] Negotiation successful
0810 03:25:40.994728 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":745,"thread_start_us":515,"threads_started":1}
W20260810 03:25:40.996541 28522 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46392
Aug 10 03:25:41 dist-test-slave-vpvm krb5kdc[28300](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332341, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 10 03:25:41 dist-test-slave-vpvm krb5kdc[28300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332341, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.161.1@KRBTEST.COM
I20260810 03:25:41.051608 29037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:41.037576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46408 (local address 127.27.161.1:33403)
0810 03:25:41.037838 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0810 03:25:41.037845 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:25:41.038116 (+ 271us) server_negotiation.cc:407] Connection header received
0810 03:25:41.038277 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:25:41.038281 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:25:41.038353 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:25:41.038439 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:25:41.039841 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:41.040532 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:41.041779 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:41.042259 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:41.042370 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:25:41.046587 (+ 4217us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:25:41.046593 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:25:41.046596 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:25:41.046638 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:25:41.049340 (+ 2702us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:25:41.050005 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:25:41.050011 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:25:41.050023 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:25:41.050165 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:25:41.050463 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:25:41.050466 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:25:41.050468 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:25:41.050714 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:25:41.050892 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0810 03:25:41.051119 (+ 227us) server_negotiation.cc:299] Negotiation successful
0810 03:25:41.051307 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260810 03:25:41.053659 28292 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iBH5/build/asan/bin/kudu pid 28458
I20260810 03:25:41.067160 29037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:41.054812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46410 (local address 127.27.161.1:33403)
0810 03:25:41.054969 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0810 03:25:41.054973 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:25:41.055363 (+ 390us) server_negotiation.cc:407] Connection header received
0810 03:25:41.055492 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:25:41.055495 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:25:41.055559 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:25:41.055634 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:25:41.056788 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:41.057469 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:41.058806 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:41.059250 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:41.059348 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:25:41.061724 (+ 2376us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:25:41.061730 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:25:41.061733 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:25:41.061770 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:25:41.064536 (+ 2766us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:25:41.065281 (+ 745us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:25:41.065286 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:25:41.065288 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:25:41.065483 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:25:41.065804 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:25:41.065808 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:25:41.065810 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:25:41.066132 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:25:41.066315 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0810 03:25:41.066572 (+ 257us) server_negotiation.cc:299] Negotiation successful
0810 03:25:41.066771 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260810 03:25:41.068109 28640 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:25:41.158066 28292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iBH5/build/asan/bin/kudu with pid 28458
I20260810 03:25:41.181877 28292 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iBH5/build/asan/bin/kudu pid 28660
Aug 10 03:25:41 dist-test-slave-vpvm krb5kdc[28300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332341, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.161.2@KRBTEST.COM
I20260810 03:25:41.198494 29045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:41.182816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49830 (local address 127.27.161.2:44723)
0810 03:25:41.183587 (+ 771us) server_negotiation.cc:206] Beginning negotiation
0810 03:25:41.183595 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 03:25:41.183623 (+ 28us) server_negotiation.cc:407] Connection header received
0810 03:25:41.183691 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:25:41.183697 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:25:41.183921 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:25:41.184052 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:25:41.185254 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:41.186364 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:41.187682 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:41.188270 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:41.188375 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:25:41.192433 (+ 4058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:25:41.192447 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:25:41.192463 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:25:41.192510 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:25:41.195474 (+ 2964us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:25:41.196139 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:25:41.196148 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:25:41.196150 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:25:41.196291 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:25:41.196722 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:25:41.196726 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:25:41.196728 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:25:41.197148 (+ 420us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:25:41.197405 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0810 03:25:41.197755 (+ 350us) server_negotiation.cc:299] Negotiation successful
0810 03:25:41.198155 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":574,"thread_start_us":409,"threads_started":1}
I20260810 03:25:41.199896 28832 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:25:41.283440 28292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iBH5/build/asan/bin/kudu with pid 28660
I20260810 03:25:41.310534 28292 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iBH5/build/asan/bin/kudu pid 28844
Aug 10 03:25:41 dist-test-slave-vpvm krb5kdc[28300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332341, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.161.3@KRBTEST.COM
I20260810 03:25:41.327272 29048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:41.311604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46048 (local address 127.27.161.3:37873)
0810 03:25:41.312169 (+ 565us) server_negotiation.cc:206] Beginning negotiation
0810 03:25:41.312177 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 03:25:41.312208 (+ 31us) server_negotiation.cc:407] Connection header received
0810 03:25:41.312270 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:25:41.312277 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:25:41.312515 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:25:41.312662 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:25:41.313994 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:41.315010 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:41.316275 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:41.316818 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:41.316934 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:25:41.321019 (+ 4085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:25:41.321031 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:25:41.321046 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:25:41.321098 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:25:41.323774 (+ 2676us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:25:41.324435 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:25:41.324442 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:25:41.324445 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:25:41.324574 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:25:41.324930 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:25:41.324934 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:25:41.324936 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:25:41.325310 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:25:41.325622 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0810 03:25:41.326725 (+ 1103us) server_negotiation.cc:299] Negotiation successful
0810 03:25:41.326948 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":404,"thread_start_us":252,"threads_started":1}
I20260810 03:25:41.328576 29015 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:25:41.411697 28292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iBH5/build/asan/bin/kudu with pid 28844
I20260810 03:25:41.437680 28292 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iBH5/build/asan/bin/kudu pid 28321
I20260810 03:25:41.438982 28408 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:25:41.545848 28292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iBH5/build/asan/bin/kudu with pid 28321
2026-08-10T03:25:41Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (5119 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786332336319544-28292-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 03:25:41 dist-test-slave-vpvm krb5kdc[29055](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 03:25:41 dist-test-slave-vpvm krb5kdc[29055](info): set up 2 sockets
Aug 10 03:25:41 dist-test-slave-vpvm krb5kdc[29055](info): commencing operation
krb5kdc: starting...
W20260810 03:25:42.763757 28292 mini_kdc.cc:121] Time spent starting KDC: real 1.166s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786332336319544-28292-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786332336319544-28292-0/krb5kdc/test-user.keytab.
Aug 10 03:25:42 dist-test-slave-vpvm krb5kdc[29055](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332342, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T03:25:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T03:25:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.161.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.161.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.161.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.62.keytab.
Entry for principal kudu/127.27.161.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.62.keytab.
I20260810 03:25:42.992188 28292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iBH5/build/asan/bin/kudu
/tmp/dist-test-task24iBH5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786332336319544-28292-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786332336319544-28292-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786332336319544-28292-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786332336319544-28292-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.161.62:41809
--webserver_interface=127.27.161.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.62.keytab
--principal=kudu/127.27.161.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.161.20:36319
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.161.62:41809
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786332336319544-28292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786332336319544-28292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786332336319544-28292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:25:43.334489 29070 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:25:43.334811 29070 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:25:43.334951 29070 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:25:43.342326 29070 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:25:43.342464 29070 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:25:43.342545 29070 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:25:43.342622 29070 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:25:43.342682 29070 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:25:43.352941 29070 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.161.20:36319
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786332336319544-28292-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786332336319544-28292-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.161.62:41809
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.62.keytab
--principal=kudu/127.27.161.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.161.62:41809
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786332336319544-28292-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.161.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786332336319544-28292-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:25:43.354410 29070 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:25:43.355769 29070 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:25:43.367821 29080 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:25:43.367779 29083 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:25:43.367858 29081 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:25:43.368818 29070 server_base.cc:1034] running on GCE node
I20260810 03:25:43.369633 29070 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:25:43.371194 29070 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:25:43.372416 29070 hybrid_clock.cc:648] HybridClock initialized: now 1786332343372363 us; error 43 us; skew 500 ppm
I20260810 03:25:43.372786 29070 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:25:43 dist-test-slave-vpvm krb5kdc[29055](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332343, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.161.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:25:43.376044 29070 init.cc:377] Logged in from keytab as kudu/127.27.161.62@KRBTEST.COM (short username kudu)
I20260810 03:25:43.377693 29070 webserver.cc:460] Webserver started at http://127.27.161.62:37203/ using document root <none> and password file <none>
I20260810 03:25:43.378481 29070 fs_manager.cc:359] Metadata directory not provided
I20260810 03:25:43.378619 29070 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:25:43.378949 29070 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:25:43.382222 29070 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786332336319544-28292-0/minicluster-data/master-0/data/instance:
uuid: "d58c8c91cb8048b99dbcdb406d46a8f4"
format_stamp: "Formatted at 2026-08-10 03:25:43 on dist-test-slave-vpvm"
server_key: "6401e74856fcf70084bab916603fc391"
server_key_iv: "bdc5db97c1004a077973bedc71f77651"
server_key_version: "encryptionkey@0"
I20260810 03:25:43.383005 29070 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786332336319544-28292-0/minicluster-data/master-0/wal/instance:
uuid: "d58c8c91cb8048b99dbcdb406d46a8f4"
format_stamp: "Formatted at 2026-08-10 03:25:43 on dist-test-slave-vpvm"
server_key: "6401e74856fcf70084bab916603fc391"
server_key_iv: "bdc5db97c1004a077973bedc71f77651"
server_key_version: "encryptionkey@0"
I20260810 03:25:43.388581 29070 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260810 03:25:43.392431 29094 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:25:43.393913 29070 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 03:25:43.394173 29070 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786332336319544-28292-0/minicluster-data/master-0/data,/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786332336319544-28292-0/minicluster-data/master-0/wal
uuid: "d58c8c91cb8048b99dbcdb406d46a8f4"
format_stamp: "Formatted at 2026-08-10 03:25:43 on dist-test-slave-vpvm"
server_key: "6401e74856fcf70084bab916603fc391"
server_key_iv: "bdc5db97c1004a077973bedc71f77651"
server_key_version: "encryptionkey@0"
I20260810 03:25:43.394389 29070 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786332336319544-28292-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786332336319544-28292-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786332336319544-28292-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:25:43.411360 29070 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:25:43.416023 29070 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:25:43.416319 29070 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:25:43.435983 29070 rpc_server.cc:307] RPC server started. Bound to: 127.27.161.62:41809
I20260810 03:25:43.436017 29178 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.161.62:41809 every 8 connection(s)
I20260810 03:25:43.437474 29070 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786332336319544-28292-0/minicluster-data/master-0/data/info.pb
I20260810 03:25:43.441569 29179 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:25:43.445487 28292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iBH5/build/asan/bin/kudu as pid 29070
I20260810 03:25:43.445740 28292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786332336319544-28292-0/minicluster-data/master-0/wal/instance
I20260810 03:25:43.446409 28292 external_mini_cluster.cc:1428] Setting key 4e2bcd627cd6dd2aae90933c4a15e9bb
I20260810 03:25:43.455790 29179 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d58c8c91cb8048b99dbcdb406d46a8f4: Bootstrap starting.
I20260810 03:25:43.459961 29179 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d58c8c91cb8048b99dbcdb406d46a8f4: Neither blocks nor log segments found. Creating new log.
I20260810 03:25:43.461436 29179 log.cc:824] T 00000000000000000000000000000000 P d58c8c91cb8048b99dbcdb406d46a8f4: Log is configured to *not* fsync() on all Append() calls
Aug 10 03:25:43 dist-test-slave-vpvm krb5kdc[29055](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332342, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.161.62@KRBTEST.COM
I20260810 03:25:43.465041 29179 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d58c8c91cb8048b99dbcdb406d46a8f4: No bootstrap required, opened a new log
I20260810 03:25:43.471271 29183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:43.448630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52778 (local address 127.27.161.62:41809)
0810 03:25:43.449465 (+ 835us) server_negotiation.cc:206] Beginning negotiation
0810 03:25:43.449477 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 03:25:43.449523 (+ 46us) server_negotiation.cc:407] Connection header received
0810 03:25:43.451291 (+ 1768us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:25:43.451319 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:25:43.451807 (+ 488us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:25:43.452176 (+ 369us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:25:43.454162 (+ 1986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:43.455622 (+ 1460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:43.457438 (+ 1816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:43.458222 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:43.458374 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:25:43.463472 (+ 5098us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:25:43.463481 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:25:43.463499 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:25:43.463540 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:25:43.466688 (+ 3148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:25:43.467450 (+ 762us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:25:43.467459 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:25:43.467466 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:25:43.467670 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:25:43.468073 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:25:43.468077 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:25:43.468079 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:25:43.468737 (+ 658us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:25:43.469053 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0810 03:25:43.469618 (+ 565us) server_negotiation.cc:299] Negotiation successful
0810 03:25:43.470122 (+ 504us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":592,"thread_start_us":362,"threads_started":1}
I20260810 03:25:43.473760 29179 raft_consensus.cc:348] T 00000000000000000000000000000000 P d58c8c91cb8048b99dbcdb406d46a8f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d58c8c91cb8048b99dbcdb406d46a8f4" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 41809 } }
I20260810 03:25:43.474298 29179 raft_consensus.cc:374] T 00000000000000000000000000000000 P d58c8c91cb8048b99dbcdb406d46a8f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:25:43.474427 29179 raft_consensus.cc:729] T 00000000000000000000000000000000 P d58c8c91cb8048b99dbcdb406d46a8f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d58c8c91cb8048b99dbcdb406d46a8f4, State: Initialized, Role: FOLLOWER
I20260810 03:25:43.475068 29179 consensus_queue.cc:260] T 00000000000000000000000000000000 P d58c8c91cb8048b99dbcdb406d46a8f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d58c8c91cb8048b99dbcdb406d46a8f4" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 41809 } }
I20260810 03:25:43.475317 29179 raft_consensus.cc:388] T 00000000000000000000000000000000 P d58c8c91cb8048b99dbcdb406d46a8f4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 03:25:43.475548 29179 raft_consensus.cc:482] T 00000000000000000000000000000000 P d58c8c91cb8048b99dbcdb406d46a8f4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 03:25:43.475745 29179 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d58c8c91cb8048b99dbcdb406d46a8f4 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:25:43.478242 29179 raft_consensus.cc:504] T 00000000000000000000000000000000 P d58c8c91cb8048b99dbcdb406d46a8f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d58c8c91cb8048b99dbcdb406d46a8f4" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 41809 } }
I20260810 03:25:43.478941 29179 leader_election.cc:302] T 00000000000000000000000000000000 P d58c8c91cb8048b99dbcdb406d46a8f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d58c8c91cb8048b99dbcdb406d46a8f4; no voters:
I20260810 03:25:43.479624 29179 leader_election.cc:288] T 00000000000000000000000000000000 P d58c8c91cb8048b99dbcdb406d46a8f4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 03:25:43.479790 29185 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d58c8c91cb8048b99dbcdb406d46a8f4 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:25:43.480502 29185 raft_consensus.cc:686] T 00000000000000000000000000000000 P d58c8c91cb8048b99dbcdb406d46a8f4 [term 1 LEADER]: Becoming Leader. State: Replica: d58c8c91cb8048b99dbcdb406d46a8f4, State: Running, Role: LEADER
I20260810 03:25:43.481096 29185 consensus_queue.cc:237] T 00000000000000000000000000000000 P d58c8c91cb8048b99dbcdb406d46a8f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d58c8c91cb8048b99dbcdb406d46a8f4" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 41809 } }
I20260810 03:25:43.481788 29179 sys_catalog.cc:564] T 00000000000000000000000000000000 P d58c8c91cb8048b99dbcdb406d46a8f4 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:25:43.485211 29193 sys_catalog.cc:455] T 00000000000000000000000000000000 P d58c8c91cb8048b99dbcdb406d46a8f4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d58c8c91cb8048b99dbcdb406d46a8f4. Latest consensus state: current_term: 1 leader_uuid: "d58c8c91cb8048b99dbcdb406d46a8f4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d58c8c91cb8048b99dbcdb406d46a8f4" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 41809 } } }
I20260810 03:25:43.485435 29191 sys_catalog.cc:455] T 00000000000000000000000000000000 P d58c8c91cb8048b99dbcdb406d46a8f4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d58c8c91cb8048b99dbcdb406d46a8f4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d58c8c91cb8048b99dbcdb406d46a8f4" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 41809 } } }
I20260810 03:25:43.485709 29193 sys_catalog.cc:458] T 00000000000000000000000000000000 P d58c8c91cb8048b99dbcdb406d46a8f4 [sys.catalog]: This master's current role is: LEADER
I20260810 03:25:43.485725 29191 sys_catalog.cc:458] T 00000000000000000000000000000000 P d58c8c91cb8048b99dbcdb406d46a8f4 [sys.catalog]: This master's current role is: LEADER
I20260810 03:25:43.487918 29202 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 03:25:43.493827 29202 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 03:25:43.504534 29202 catalog_manager.cc:1347] Generated new cluster ID: 3256878d0d754d108cb9bfa2e27b3212
I20260810 03:25:43.504670 29202 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 03:25:43.527172 29202 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 03:25:43.528553 29202 catalog_manager.cc:1504] Loading token signing keys...
I20260810 03:25:43.535961 29202 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d58c8c91cb8048b99dbcdb406d46a8f4: Generated new TSK 0
I20260810 03:25:43.537062 29202 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.161.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.161.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.161.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.1.keytab.
Entry for principal kudu/127.27.161.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.1.keytab.
I20260810 03:25:43.605727 28292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iBH5/build/asan/bin/kudu
/tmp/dist-test-task24iBH5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786332336319544-28292-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786332336319544-28292-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786332336319544-28292-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786332336319544-28292-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.161.1:0
--local_ip_for_outbound_sockets=127.27.161.1
--webserver_interface=127.27.161.1
--webserver_port=0
--tserver_master_addrs=127.27.161.62:41809
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.1.keytab
--principal=kudu/127.27.161.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.161.20:36319
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786332336319544-28292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786332336319544-28292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786332336319544-28292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:25:43.980548 29218 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:25:43.980852 29218 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:25:43.980988 29218 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:25:43.988627 29218 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:25:43.988742 29218 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:25:43.988893 29218 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.161.1
I20260810 03:25:43.999018 29218 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.161.20:36319
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786332336319544-28292-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786332336319544-28292-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.1.keytab
--principal=kudu/127.27.161.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.161.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786332336319544-28292-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.161.1
--webserver_port=0
--tserver_master_addrs=127.27.161.62:41809
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.161.1
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786332336319544-28292-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:25:44.000440 29218 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:25:44.001842 29218 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:25:44.015182 29234 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:25:44.015218 29229 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:25:44.015142 29226 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:25:44.016247 29218 server_base.cc:1034] running on GCE node
I20260810 03:25:44.016896 29218 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:25:44.017874 29218 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:25:44.019102 29218 hybrid_clock.cc:648] HybridClock initialized: now 1786332344019052 us; error 45 us; skew 500 ppm
I20260810 03:25:44.019418 29218 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:25:44 dist-test-slave-vpvm krb5kdc[29055](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332344, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.161.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:25:44.025745 29218 init.cc:377] Logged in from keytab as kudu/127.27.161.1@KRBTEST.COM (short username kudu)
I20260810 03:25:44.027587 29218 webserver.cc:460] Webserver started at http://127.27.161.1:33729/ using document root <none> and password file <none>
I20260810 03:25:44.028344 29218 fs_manager.cc:359] Metadata directory not provided
I20260810 03:25:44.028471 29218 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:25:44.028807 29218 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:25:44.031952 29218 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786332336319544-28292-0/minicluster-data/ts-0/data/instance:
uuid: "ceece70b4fe7465f9ae0c9f3299ffa7f"
format_stamp: "Formatted at 2026-08-10 03:25:44 on dist-test-slave-vpvm"
server_key: "4e87938bd1b104fec9ef628903ca998b"
server_key_iv: "3743ad3be46ed9c3ab6bbb958934d976"
server_key_version: "encryptionkey@0"
I20260810 03:25:44.032778 29218 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786332336319544-28292-0/minicluster-data/ts-0/wal/instance:
uuid: "ceece70b4fe7465f9ae0c9f3299ffa7f"
format_stamp: "Formatted at 2026-08-10 03:25:44 on dist-test-slave-vpvm"
server_key: "4e87938bd1b104fec9ef628903ca998b"
server_key_iv: "3743ad3be46ed9c3ab6bbb958934d976"
server_key_version: "encryptionkey@0"
I20260810 03:25:44.038328 29218 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260810 03:25:44.042135 29245 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:25:44.043560 29218 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 03:25:44.043735 29218 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786332336319544-28292-0/minicluster-data/ts-0/data,/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786332336319544-28292-0/minicluster-data/ts-0/wal
uuid: "ceece70b4fe7465f9ae0c9f3299ffa7f"
format_stamp: "Formatted at 2026-08-10 03:25:44 on dist-test-slave-vpvm"
server_key: "4e87938bd1b104fec9ef628903ca998b"
server_key_iv: "3743ad3be46ed9c3ab6bbb958934d976"
server_key_version: "encryptionkey@0"
I20260810 03:25:44.043944 29218 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786332336319544-28292-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786332336319544-28292-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786332336319544-28292-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:25:44.066635 29218 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:25:44.070446 29218 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:25:44.070719 29218 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:25:44.071718 29218 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:25:44.073076 29218 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:25:44.073158 29218 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:25:44.073283 29218 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:25:44.073339 29218 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:25:44.105463 29218 rpc_server.cc:307] RPC server started. Bound to: 127.27.161.1:42023
I20260810 03:25:44.105508 29403 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.161.1:42023 every 8 connection(s)
I20260810 03:25:44.106967 29218 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786332336319544-28292-0/minicluster-data/ts-0/data/info.pb
I20260810 03:25:44.112299 28292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iBH5/build/asan/bin/kudu as pid 29218
I20260810 03:25:44.112509 28292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786332336319544-28292-0/minicluster-data/ts-0/wal/instance
I20260810 03:25:44.113123 28292 external_mini_cluster.cc:1428] Setting key 64adb9a1fb9b2ed4e3c548a329e0b3a1
Aug 10 03:25:44 dist-test-slave-vpvm krb5kdc[29055](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332344, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.161.1@KRBTEST.COM for kudu/127.27.161.62@KRBTEST.COM
I20260810 03:25:44.129106 29408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:44.109660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.161.1:44063 (local address 127.27.161.62:41809)
0810 03:25:44.110190 (+ 530us) server_negotiation.cc:206] Beginning negotiation
0810 03:25:44.110196 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:25:44.111093 (+ 897us) server_negotiation.cc:407] Connection header received
0810 03:25:44.113178 (+ 2085us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:25:44.113185 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:25:44.113283 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:25:44.113394 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:25:44.115176 (+ 1782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:44.115926 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:44.117652 (+ 1726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:44.118155 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:44.118297 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:25:44.122208 (+ 3911us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:25:44.122216 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:25:44.122221 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:25:44.122268 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:25:44.125473 (+ 3205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:25:44.126486 (+ 1013us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:25:44.126491 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:25:44.126494 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:25:44.126656 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:25:44.127186 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:25:44.127191 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:25:44.127194 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:25:44.127537 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:25:44.127679 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0810 03:25:44.128508 (+ 829us) server_negotiation.cc:299] Negotiation successful
0810 03:25:44.128744 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":414,"thread_start_us":287,"threads_started":1}
I20260810 03:25:44.130210 29411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:44.110090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:41809 (local address 127.27.161.1:44063)
0810 03:25:44.110880 (+ 790us) negotiation.cc:107] Waiting for socket to connect
0810 03:25:44.110936 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0810 03:25:44.112858 (+ 1922us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:25:44.113680 (+ 822us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:25:44.113690 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:25:44.114339 (+ 649us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:25:44.114984 (+ 645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:25:44.115006 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:25:44.116130 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:25:44.116134 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:25:44.117474 (+ 1340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:25:44.117492 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:25:44.118420 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:25:44.118429 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:25:44.118699 (+ 270us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:25:44.119423 (+ 724us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:25:44.119451 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:25:44.121887 (+ 2436us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:25:44.125684 (+ 3797us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:25:44.125693 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:25:44.125710 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:25:44.126317 (+ 607us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:25:44.126796 (+ 479us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:25:44.126800 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:25:44.126803 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:25:44.127054 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:25:44.127683 (+ 629us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:25:44.127690 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:25:44.128081 (+ 391us) client_negotiation.cc:769] Sending connection context
0810 03:25:44.128349 (+ 268us) client_negotiation.cc:240] Negotiation successful
0810 03:25:44.128828 (+ 479us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":533,"thread_start_us":296,"threads_started":1}
I20260810 03:25:44.132189 29404 heartbeater.cc:347] Connected to a master server at 127.27.161.62:41809
I20260810 03:25:44.132577 29404 heartbeater.cc:464] Registering TS with master...
I20260810 03:25:44.133659 29404 heartbeater.cc:511] Master 127.27.161.62:41809 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.27.161.2@KRBTEST.COM; defaulting to no policy
I20260810 03:25:44.136315 29115 ts_manager.cc:194] Registered new tserver with Master: ceece70b4fe7465f9ae0c9f3299ffa7f (127.27.161.1:42023)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.161.2@KRBTEST.COM" created.
I20260810 03:25:44.138314 29115 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.161.1@KRBTEST.COM'} at 127.27.161.1:44063
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.161.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.2.keytab.
Entry for principal kudu/127.27.161.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.2.keytab.
I20260810 03:25:44.165760 28292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iBH5/build/asan/bin/kudu
/tmp/dist-test-task24iBH5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786332336319544-28292-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786332336319544-28292-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786332336319544-28292-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786332336319544-28292-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.161.2:0
--local_ip_for_outbound_sockets=127.27.161.2
--webserver_interface=127.27.161.2
--webserver_port=0
--tserver_master_addrs=127.27.161.62:41809
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.2.keytab
--principal=kudu/127.27.161.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.161.20:36319
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786332336319544-28292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786332336319544-28292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786332336319544-28292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:25:44.512850 29418 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:25:44.513180 29418 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:25:44.513334 29418 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:25:44.520821 29418 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:25:44.520959 29418 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:25:44.521126 29418 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.161.2
I20260810 03:25:44.531391 29418 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.161.20:36319
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786332336319544-28292-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786332336319544-28292-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.2.keytab
--principal=kudu/127.27.161.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.161.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786332336319544-28292-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.161.2
--webserver_port=0
--tserver_master_addrs=127.27.161.62:41809
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.161.2
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786332336319544-28292-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:25:44.532737 29418 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:25:44.534137 29418 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:25:44.547459 29424 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:25:44.547593 29427 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:25:44.547621 29425 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:25:44.548875 29418 server_base.cc:1034] running on GCE node
I20260810 03:25:44.549562 29418 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:25:44.550631 29418 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:25:44.551846 29418 hybrid_clock.cc:648] HybridClock initialized: now 1786332344551788 us; error 45 us; skew 500 ppm
I20260810 03:25:44.552174 29418 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:25:44 dist-test-slave-vpvm krb5kdc[29055](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332344, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.161.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:25:44.555450 29418 init.cc:377] Logged in from keytab as kudu/127.27.161.2@KRBTEST.COM (short username kudu)
I20260810 03:25:44.557196 29418 webserver.cc:460] Webserver started at http://127.27.161.2:40719/ using document root <none> and password file <none>
I20260810 03:25:44.557920 29418 fs_manager.cc:359] Metadata directory not provided
I20260810 03:25:44.558070 29418 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:25:44.558368 29418 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:25:44.561426 29418 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786332336319544-28292-0/minicluster-data/ts-1/data/instance:
uuid: "436223292ce14a96b6da9826e04462b2"
format_stamp: "Formatted at 2026-08-10 03:25:44 on dist-test-slave-vpvm"
server_key: "243307b49ea5baccce60335ed638f96c"
server_key_iv: "262916c9478e9bfec862933dfe21f0ac"
server_key_version: "encryptionkey@0"
I20260810 03:25:44.562392 29418 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786332336319544-28292-0/minicluster-data/ts-1/wal/instance:
uuid: "436223292ce14a96b6da9826e04462b2"
format_stamp: "Formatted at 2026-08-10 03:25:44 on dist-test-slave-vpvm"
server_key: "243307b49ea5baccce60335ed638f96c"
server_key_iv: "262916c9478e9bfec862933dfe21f0ac"
server_key_version: "encryptionkey@0"
I20260810 03:25:44.568262 29418 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260810 03:25:44.572239 29438 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:25:44.573746 29418 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 03:25:44.573930 29418 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786332336319544-28292-0/minicluster-data/ts-1/data,/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786332336319544-28292-0/minicluster-data/ts-1/wal
uuid: "436223292ce14a96b6da9826e04462b2"
format_stamp: "Formatted at 2026-08-10 03:25:44 on dist-test-slave-vpvm"
server_key: "243307b49ea5baccce60335ed638f96c"
server_key_iv: "262916c9478e9bfec862933dfe21f0ac"
server_key_version: "encryptionkey@0"
I20260810 03:25:44.574185 29418 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786332336319544-28292-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786332336319544-28292-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786332336319544-28292-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:25:44.588223 29418 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:25:44.593021 29418 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:25:44.593330 29418 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:25:44.594483 29418 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:25:44.595939 29418 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:25:44.596016 29418 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:25:44.596136 29418 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:25:44.596189 29418 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:25:44.629014 29418 rpc_server.cc:307] RPC server started. Bound to: 127.27.161.2:35291
I20260810 03:25:44.629066 29606 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.161.2:35291 every 8 connection(s)
I20260810 03:25:44.630352 29418 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786332336319544-28292-0/minicluster-data/ts-1/data/info.pb
I20260810 03:25:44.638432 28292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iBH5/build/asan/bin/kudu as pid 29418
I20260810 03:25:44.638616 28292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786332336319544-28292-0/minicluster-data/ts-1/wal/instance
I20260810 03:25:44.639070 28292 external_mini_cluster.cc:1428] Setting key 0e192d9eb48f90e6e44a1974fc12d346
Aug 10 03:25:44 dist-test-slave-vpvm krb5kdc[29055](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332344, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.161.2@KRBTEST.COM for kudu/127.27.161.62@KRBTEST.COM
I20260810 03:25:44.651741 29610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:44.632986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.161.2:51801 (local address 127.27.161.62:41809)
0810 03:25:44.633403 (+ 417us) server_negotiation.cc:206] Beginning negotiation
0810 03:25:44.633407 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:25:44.634500 (+ 1093us) server_negotiation.cc:407] Connection header received
0810 03:25:44.636353 (+ 1853us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:25:44.636359 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:25:44.636447 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:25:44.636552 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:25:44.638212 (+ 1660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:44.639015 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:44.640808 (+ 1793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:44.641256 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:44.641378 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:25:44.645304 (+ 3926us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:25:44.645310 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:25:44.645314 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:25:44.645350 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:25:44.648106 (+ 2756us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:25:44.649093 (+ 987us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:25:44.649099 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:25:44.649103 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:25:44.649271 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:25:44.649711 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:25:44.649716 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:25:44.649719 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:25:44.650137 (+ 418us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:25:44.650291 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0810 03:25:44.651130 (+ 839us) server_negotiation.cc:299] Negotiation successful
0810 03:25:44.651373 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307,"thread_start_us":204,"threads_started":1}
I20260810 03:25:44.652667 29616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:44.633534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:41809 (local address 127.27.161.2:51801)
0810 03:25:44.634325 (+ 791us) negotiation.cc:107] Waiting for socket to connect
0810 03:25:44.634375 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0810 03:25:44.636072 (+ 1697us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:25:44.636803 (+ 731us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:25:44.636812 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:25:44.637353 (+ 541us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:25:44.637935 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:25:44.638000 (+ 65us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:25:44.639187 (+ 1187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:25:44.639193 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:25:44.640656 (+ 1463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:25:44.640670 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:25:44.641500 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:25:44.641508 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:25:44.641736 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:25:44.642481 (+ 745us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:25:44.642508 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:25:44.645000 (+ 2492us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:25:44.648275 (+ 3275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:25:44.648286 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:25:44.648305 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:25:44.648944 (+ 639us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:25:44.649373 (+ 429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:25:44.649376 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:25:44.649378 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:25:44.649578 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:25:44.650302 (+ 724us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:25:44.650310 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:25:44.650703 (+ 393us) client_negotiation.cc:769] Sending connection context
0810 03:25:44.650973 (+ 270us) client_negotiation.cc:240] Negotiation successful
0810 03:25:44.651443 (+ 470us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":552,"thread_start_us":268,"threads_started":1}
I20260810 03:25:44.654140 29607 heartbeater.cc:347] Connected to a master server at 127.27.161.62:41809
I20260810 03:25:44.654461 29607 heartbeater.cc:464] Registering TS with master...
I20260810 03:25:44.655246 29607 heartbeater.cc:511] Master 127.27.161.62:41809 requested a full tablet report, sending...
I20260810 03:25:44.656810 29115 ts_manager.cc:194] Registered new tserver with Master: 436223292ce14a96b6da9826e04462b2 (127.27.161.2:35291)
I20260810 03:25:44.657670 29115 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.161.2@KRBTEST.COM'} at 127.27.161.2:51801
WARNING: no policy specified for kudu/127.27.161.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.161.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.161.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.3.keytab.
Entry for principal kudu/127.27.161.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.3.keytab.
I20260810 03:25:44.689213 28292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iBH5/build/asan/bin/kudu
/tmp/dist-test-task24iBH5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786332336319544-28292-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786332336319544-28292-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786332336319544-28292-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786332336319544-28292-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.161.3:0
--local_ip_for_outbound_sockets=127.27.161.3
--webserver_interface=127.27.161.3
--webserver_port=0
--tserver_master_addrs=127.27.161.62:41809
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.3.keytab
--principal=kudu/127.27.161.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.161.20:36319
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786332336319544-28292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786332336319544-28292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786332336319544-28292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:25:45.028486 29619 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:25:45.028820 29619 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:25:45.028986 29619 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:25:45.036402 29619 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:25:45.036526 29619 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:25:45.036650 29619 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.161.3
I20260810 03:25:45.046798 29619 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.161.20:36319
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786332336319544-28292-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786332336319544-28292-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.3.keytab
--principal=kudu/127.27.161.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.161.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786332336319544-28292-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.161.3
--webserver_port=0
--tserver_master_addrs=127.27.161.62:41809
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.161.3
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786332336319544-28292-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:25:45.048224 29619 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:25:45.049706 29619 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:25:45.063054 29625 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:25:45.063102 29626 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:25:45.063073 29629 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:25:45.063959 29619 server_base.cc:1034] running on GCE node
I20260810 03:25:45.064548 29619 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:25:45.065502 29619 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:25:45.066716 29619 hybrid_clock.cc:648] HybridClock initialized: now 1786332345066672 us; error 34 us; skew 500 ppm
I20260810 03:25:45.067014 29619 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:25:45 dist-test-slave-vpvm krb5kdc[29055](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332345, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.161.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:25:45.070156 29619 init.cc:377] Logged in from keytab as kudu/127.27.161.3@KRBTEST.COM (short username kudu)
I20260810 03:25:45.071789 29619 webserver.cc:460] Webserver started at http://127.27.161.3:36149/ using document root <none> and password file <none>
I20260810 03:25:45.072487 29619 fs_manager.cc:359] Metadata directory not provided
I20260810 03:25:45.072590 29619 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:25:45.072878 29619 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:25:45.075963 29619 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786332336319544-28292-0/minicluster-data/ts-2/data/instance:
uuid: "f272843f51044e3ab3fe3234eaaabbc2"
format_stamp: "Formatted at 2026-08-10 03:25:45 on dist-test-slave-vpvm"
server_key: "1452032e320849277f6a13b80fc7f114"
server_key_iv: "8b52c6ccdc1e75a38639f62e10aaf3b6"
server_key_version: "encryptionkey@0"
I20260810 03:25:45.076773 29619 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786332336319544-28292-0/minicluster-data/ts-2/wal/instance:
uuid: "f272843f51044e3ab3fe3234eaaabbc2"
format_stamp: "Formatted at 2026-08-10 03:25:45 on dist-test-slave-vpvm"
server_key: "1452032e320849277f6a13b80fc7f114"
server_key_iv: "8b52c6ccdc1e75a38639f62e10aaf3b6"
server_key_version: "encryptionkey@0"
I20260810 03:25:45.082975 29619 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.004s
I20260810 03:25:45.086917 29640 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:25:45.088392 29619 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 03:25:45.088553 29619 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786332336319544-28292-0/minicluster-data/ts-2/data,/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786332336319544-28292-0/minicluster-data/ts-2/wal
uuid: "f272843f51044e3ab3fe3234eaaabbc2"
format_stamp: "Formatted at 2026-08-10 03:25:45 on dist-test-slave-vpvm"
server_key: "1452032e320849277f6a13b80fc7f114"
server_key_iv: "8b52c6ccdc1e75a38639f62e10aaf3b6"
server_key_version: "encryptionkey@0"
I20260810 03:25:45.088757 29619 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786332336319544-28292-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786332336319544-28292-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786332336319544-28292-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:25:45.114157 29619 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:25:45.118903 29619 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:25:45.119199 29619 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:25:45.120285 29619 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:25:45.121888 29619 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:25:45.122004 29619 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260810 03:25:45.122126 29619 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:25:45.122180 29619 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:25:45.142809 29404 heartbeater.cc:503] Master 127.27.161.62:41809 was elected leader, sending a full tablet report...
I20260810 03:25:45.160799 29619 rpc_server.cc:307] RPC server started. Bound to: 127.27.161.3:32851
I20260810 03:25:45.160851 29802 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.161.3:32851 every 8 connection(s)
I20260810 03:25:45.162233 29619 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786332336319544-28292-0/minicluster-data/ts-2/data/info.pb
I20260810 03:25:45.171902 28292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iBH5/build/asan/bin/kudu as pid 29619
I20260810 03:25:45.172087 28292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786332336319544-28292-0/minicluster-data/ts-2/wal/instance
I20260810 03:25:45.172521 28292 external_mini_cluster.cc:1428] Setting key 3e7829041822630d5540399225eddb3e
Aug 10 03:25:45 dist-test-slave-vpvm krb5kdc[29055](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332345, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.161.3@KRBTEST.COM for kudu/127.27.161.62@KRBTEST.COM
I20260810 03:25:45.184659 29809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:45.165027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.161.3:58227 (local address 127.27.161.62:41809)
0810 03:25:45.165835 (+ 808us) server_negotiation.cc:206] Beginning negotiation
0810 03:25:45.165840 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:25:45.166710 (+ 870us) server_negotiation.cc:407] Connection header received
0810 03:25:45.168945 (+ 2235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:25:45.168949 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:25:45.169037 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:25:45.169234 (+ 197us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:25:45.171082 (+ 1848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:45.172103 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:45.174143 (+ 2040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:45.174726 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:45.174860 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:25:45.178710 (+ 3850us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:25:45.178716 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:25:45.178719 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:25:45.178756 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:25:45.181476 (+ 2720us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:25:45.182402 (+ 926us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:25:45.182408 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:25:45.182412 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:25:45.182567 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:25:45.182991 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:25:45.182994 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:25:45.182996 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:25:45.183314 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:25:45.183455 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0810 03:25:45.184190 (+ 735us) server_negotiation.cc:299] Negotiation successful
0810 03:25:45.184374 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":702,"thread_start_us":580,"threads_started":1}
I20260810 03:25:45.185756 29812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:45.165467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:41809 (local address 127.27.161.3:58227)
0810 03:25:45.166467 (+ 1000us) negotiation.cc:107] Waiting for socket to connect
0810 03:25:45.166548 (+ 81us) client_negotiation.cc:174] Beginning negotiation
0810 03:25:45.168644 (+ 2096us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:25:45.169447 (+ 803us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:25:45.169455 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:25:45.170185 (+ 730us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:25:45.170887 (+ 702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:25:45.170909 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:25:45.172277 (+ 1368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:25:45.172281 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:25:45.173984 (+ 1703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:25:45.174005 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:25:45.174967 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:25:45.174977 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:25:45.175296 (+ 319us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:25:45.176071 (+ 775us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:25:45.176097 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:25:45.178477 (+ 2380us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:25:45.181653 (+ 3176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:25:45.181662 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:25:45.181678 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:25:45.182263 (+ 585us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:25:45.182683 (+ 420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:25:45.182687 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:25:45.182689 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:25:45.182898 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:25:45.183465 (+ 567us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:25:45.183471 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:25:45.183836 (+ 365us) client_negotiation.cc:769] Sending connection context
0810 03:25:45.184082 (+ 246us) client_negotiation.cc:240] Negotiation successful
0810 03:25:45.184507 (+ 425us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":648,"thread_start_us":427,"threads_started":1}
I20260810 03:25:45.187731 29803 heartbeater.cc:347] Connected to a master server at 127.27.161.62:41809
I20260810 03:25:45.188195 29803 heartbeater.cc:464] Registering TS with master...
I20260810 03:25:45.189139 29803 heartbeater.cc:511] Master 127.27.161.62:41809 requested a full tablet report, sending...
I20260810 03:25:45.190881 29115 ts_manager.cc:194] Registered new tserver with Master: f272843f51044e3ab3fe3234eaaabbc2 (127.27.161.3:32851)
I20260810 03:25:45.192003 29115 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.161.3@KRBTEST.COM'} at 127.27.161.3:58227
I20260810 03:25:45.201126 28292 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 03:25:45.224383 29809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:45.209216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52780 (local address 127.27.161.62:41809)
0810 03:25:45.209490 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0810 03:25:45.209499 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 03:25:45.210211 (+ 712us) server_negotiation.cc:407] Connection header received
0810 03:25:45.210327 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:25:45.210330 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:25:45.210413 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:25:45.210520 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:25:45.212199 (+ 1679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:45.212955 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:45.214379 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:45.214890 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:45.215000 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:25:45.218562 (+ 3562us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:25:45.218568 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:25:45.218572 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:25:45.218611 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:25:45.221503 (+ 2892us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:25:45.222435 (+ 932us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:25:45.222438 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:25:45.222441 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:25:45.222567 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:25:45.223026 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:25:45.223030 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:25:45.223032 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:25:45.223411 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:25:45.223604 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0810 03:25:45.223909 (+ 305us) server_negotiation.cc:299] Negotiation successful
0810 03:25:45.224091 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260810 03:25:45.233608 29115 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52780:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 03:25:45.243639 29115 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 03:25:45.286986 29829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:45.260815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.3:32851 (local address 127.0.0.1:55672)
0810 03:25:45.276228 (+ 15413us) negotiation.cc:107] Waiting for socket to connect
0810 03:25:45.276250 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0810 03:25:45.276494 (+ 244us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:25:45.277980 (+ 1486us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:25:45.277989 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:25:45.278025 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:25:45.278793 (+ 768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:25:45.278811 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:25:45.280926 (+ 2115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:25:45.280931 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:25:45.283776 (+ 2845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:25:45.283789 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:25:45.285663 (+ 1874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:25:45.285673 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:25:45.286036 (+ 363us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:25:45.286081 (+ 45us) client_negotiation.cc:769] Sending connection context
0810 03:25:45.286278 (+ 197us) client_negotiation.cc:240] Negotiation successful
0810 03:25:45.286500 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15207,"mutex_wait_us":58,"thread_start_us":12711,"threads_started":1}
I20260810 03:25:45.286988 29835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:45.261595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55672 (local address 127.27.161.3:32851)
0810 03:25:45.275368 (+ 13773us) server_negotiation.cc:206] Beginning negotiation
0810 03:25:45.275380 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 03:25:45.276363 (+ 983us) server_negotiation.cc:407] Connection header received
0810 03:25:45.276763 (+ 400us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:25:45.276773 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:25:45.277119 (+ 346us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:25:45.277431 (+ 312us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:25:45.279031 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:45.280767 (+ 1736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:45.283972 (+ 3205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:45.285413 (+ 1441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:45.285549 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:25:45.285818 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0810 03:25:45.286374 (+ 556us) server_negotiation.cc:299] Negotiation successful
0810 03:25:45.286571 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13557,"thread_start_us":13376,"threads_started":1}
I20260810 03:25:45.288990 29830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:45.260657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.2:35291 (local address 127.0.0.1:43188)
0810 03:25:45.274009 (+ 13352us) negotiation.cc:107] Waiting for socket to connect
0810 03:25:45.274057 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0810 03:25:45.274490 (+ 433us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:25:45.281520 (+ 7030us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:25:45.281525 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:25:45.281550 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:25:45.282164 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:25:45.282174 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:25:45.283864 (+ 1690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:25:45.283868 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:25:45.286049 (+ 2181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:25:45.286065 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:25:45.288164 (+ 2099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:25:45.288171 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:25:45.288348 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:25:45.288382 (+ 34us) client_negotiation.cc:769] Sending connection context
0810 03:25:45.288446 (+ 64us) client_negotiation.cc:240] Negotiation successful
0810 03:25:45.288594 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13036,"mutex_wait_us":70,"thread_start_us":14987,"threads_started":1}
I20260810 03:25:45.289927 29833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:45.261240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.1:42023 (local address 127.0.0.1:45010)
0810 03:25:45.279610 (+ 18370us) negotiation.cc:107] Waiting for socket to connect
0810 03:25:45.279629 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 03:25:45.279843 (+ 214us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:25:45.280979 (+ 1136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:25:45.280983 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:25:45.281012 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:25:45.281627 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:25:45.281641 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:25:45.284620 (+ 2979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:25:45.284626 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:25:45.286894 (+ 2268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:25:45.286910 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:25:45.289088 (+ 2178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:25:45.289094 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:25:45.289264 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:25:45.289294 (+ 30us) client_negotiation.cc:769] Sending connection context
0810 03:25:45.289375 (+ 81us) client_negotiation.cc:240] Negotiation successful
0810 03:25:45.289506 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18229,"mutex_wait_us":69,"thread_start_us":18084,"threads_started":1}
I20260810 03:25:45.289928 29839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:45.261848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45010 (local address 127.27.161.1:42023)
0810 03:25:45.280262 (+ 18414us) server_negotiation.cc:206] Beginning negotiation
0810 03:25:45.280271 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 03:25:45.280301 (+ 30us) server_negotiation.cc:407] Connection header received
0810 03:25:45.280391 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:25:45.280399 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:25:45.280647 (+ 248us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:25:45.280952 (+ 305us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:25:45.282011 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:45.284468 (+ 2457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:45.287110 (+ 2642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:45.288859 (+ 1749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:45.288992 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:25:45.289227 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0810 03:25:45.289437 (+ 210us) server_negotiation.cc:299] Negotiation successful
0810 03:25:45.289627 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18224,"thread_start_us":18089,"threads_started":1}
I20260810 03:25:45.290411 29831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:45.260794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43188 (local address 127.27.161.2:35291)
0810 03:25:45.280821 (+ 20027us) server_negotiation.cc:206] Beginning negotiation
0810 03:25:45.280827 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:25:45.280850 (+ 23us) server_negotiation.cc:407] Connection header received
0810 03:25:45.280946 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:25:45.280954 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:25:45.281186 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:25:45.281530 (+ 344us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:25:45.282323 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:45.283724 (+ 1401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:45.286230 (+ 2506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:45.287925 (+ 1695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:45.289406 (+ 1481us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:25:45.289610 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0810 03:25:45.289703 (+ 93us) server_negotiation.cc:299] Negotiation successful
0810 03:25:45.289882 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19873,"thread_start_us":19753,"threads_started":1}
I20260810 03:25:45.292277 29709 tablet_service.cc:1467] Processing CreateTablet for tablet ad79026708c64d1b863119191198722e (DEFAULT_TABLE table=test-table [id=8d135f983ef2498eb41874940fbff10e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:25:45.293352 29312 tablet_service.cc:1467] Processing CreateTablet for tablet ad79026708c64d1b863119191198722e (DEFAULT_TABLE table=test-table [id=8d135f983ef2498eb41874940fbff10e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:25:45.294147 29709 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad79026708c64d1b863119191198722e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:25:45.294538 29312 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad79026708c64d1b863119191198722e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:25:45.294291 29514 tablet_service.cc:1467] Processing CreateTablet for tablet ad79026708c64d1b863119191198722e (DEFAULT_TABLE table=test-table [id=8d135f983ef2498eb41874940fbff10e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:25:45.295668 29514 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad79026708c64d1b863119191198722e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:25:45.307771 29841 tablet_bootstrap.cc:492] T ad79026708c64d1b863119191198722e P ceece70b4fe7465f9ae0c9f3299ffa7f: Bootstrap starting.
I20260810 03:25:45.312299 29841 tablet_bootstrap.cc:654] T ad79026708c64d1b863119191198722e P ceece70b4fe7465f9ae0c9f3299ffa7f: Neither blocks nor log segments found. Creating new log.
I20260810 03:25:45.312619 29843 tablet_bootstrap.cc:492] T ad79026708c64d1b863119191198722e P f272843f51044e3ab3fe3234eaaabbc2: Bootstrap starting.
I20260810 03:25:45.313843 29841 log.cc:824] T ad79026708c64d1b863119191198722e P ceece70b4fe7465f9ae0c9f3299ffa7f: Log is configured to *not* fsync() on all Append() calls
I20260810 03:25:45.314394 29845 tablet_bootstrap.cc:492] T ad79026708c64d1b863119191198722e P 436223292ce14a96b6da9826e04462b2: Bootstrap starting.
I20260810 03:25:45.316514 29843 tablet_bootstrap.cc:654] T ad79026708c64d1b863119191198722e P f272843f51044e3ab3fe3234eaaabbc2: Neither blocks nor log segments found. Creating new log.
I20260810 03:25:45.317876 29843 log.cc:824] T ad79026708c64d1b863119191198722e P f272843f51044e3ab3fe3234eaaabbc2: Log is configured to *not* fsync() on all Append() calls
I20260810 03:25:45.318261 29845 tablet_bootstrap.cc:654] T ad79026708c64d1b863119191198722e P 436223292ce14a96b6da9826e04462b2: Neither blocks nor log segments found. Creating new log.
I20260810 03:25:45.318292 29841 tablet_bootstrap.cc:492] T ad79026708c64d1b863119191198722e P ceece70b4fe7465f9ae0c9f3299ffa7f: No bootstrap required, opened a new log
I20260810 03:25:45.318591 29841 ts_tablet_manager.cc:1397] T ad79026708c64d1b863119191198722e P ceece70b4fe7465f9ae0c9f3299ffa7f: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.008s
I20260810 03:25:45.319705 29845 log.cc:824] T ad79026708c64d1b863119191198722e P 436223292ce14a96b6da9826e04462b2: Log is configured to *not* fsync() on all Append() calls
I20260810 03:25:45.320678 29843 tablet_bootstrap.cc:492] T ad79026708c64d1b863119191198722e P f272843f51044e3ab3fe3234eaaabbc2: No bootstrap required, opened a new log
I20260810 03:25:45.321007 29843 ts_tablet_manager.cc:1397] T ad79026708c64d1b863119191198722e P f272843f51044e3ab3fe3234eaaabbc2: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.006s
I20260810 03:25:45.323050 29845 tablet_bootstrap.cc:492] T ad79026708c64d1b863119191198722e P 436223292ce14a96b6da9826e04462b2: No bootstrap required, opened a new log
I20260810 03:25:45.323346 29845 ts_tablet_manager.cc:1397] T ad79026708c64d1b863119191198722e P 436223292ce14a96b6da9826e04462b2: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.004s
I20260810 03:25:45.329356 29841 raft_consensus.cc:348] T ad79026708c64d1b863119191198722e P ceece70b4fe7465f9ae0c9f3299ffa7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "436223292ce14a96b6da9826e04462b2" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 35291 } } peers { permanent_uuid: "f272843f51044e3ab3fe3234eaaabbc2" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 32851 } } peers { permanent_uuid: "ceece70b4fe7465f9ae0c9f3299ffa7f" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 42023 } }
I20260810 03:25:45.330050 29841 raft_consensus.cc:374] T ad79026708c64d1b863119191198722e P ceece70b4fe7465f9ae0c9f3299ffa7f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:25:45.330193 29841 raft_consensus.cc:729] T ad79026708c64d1b863119191198722e P ceece70b4fe7465f9ae0c9f3299ffa7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ceece70b4fe7465f9ae0c9f3299ffa7f, State: Initialized, Role: FOLLOWER
I20260810 03:25:45.330017 29843 raft_consensus.cc:348] T ad79026708c64d1b863119191198722e P f272843f51044e3ab3fe3234eaaabbc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "436223292ce14a96b6da9826e04462b2" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 35291 } } peers { permanent_uuid: "f272843f51044e3ab3fe3234eaaabbc2" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 32851 } } peers { permanent_uuid: "ceece70b4fe7465f9ae0c9f3299ffa7f" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 42023 } }
I20260810 03:25:45.330487 29843 raft_consensus.cc:374] T ad79026708c64d1b863119191198722e P f272843f51044e3ab3fe3234eaaabbc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:25:45.330612 29843 raft_consensus.cc:729] T ad79026708c64d1b863119191198722e P f272843f51044e3ab3fe3234eaaabbc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f272843f51044e3ab3fe3234eaaabbc2, State: Initialized, Role: FOLLOWER
I20260810 03:25:45.330986 29841 consensus_queue.cc:260] T ad79026708c64d1b863119191198722e P ceece70b4fe7465f9ae0c9f3299ffa7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "436223292ce14a96b6da9826e04462b2" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 35291 } } peers { permanent_uuid: "f272843f51044e3ab3fe3234eaaabbc2" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 32851 } } peers { permanent_uuid: "ceece70b4fe7465f9ae0c9f3299ffa7f" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 42023 } }
I20260810 03:25:45.331234 29843 consensus_queue.cc:260] T ad79026708c64d1b863119191198722e P f272843f51044e3ab3fe3234eaaabbc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "436223292ce14a96b6da9826e04462b2" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 35291 } } peers { permanent_uuid: "f272843f51044e3ab3fe3234eaaabbc2" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 32851 } } peers { permanent_uuid: "ceece70b4fe7465f9ae0c9f3299ffa7f" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 42023 } }
I20260810 03:25:45.333107 29841 ts_tablet_manager.cc:1428] T ad79026708c64d1b863119191198722e P ceece70b4fe7465f9ae0c9f3299ffa7f: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260810 03:25:45.334056 29803 heartbeater.cc:503] Master 127.27.161.62:41809 was elected leader, sending a full tablet report...
I20260810 03:25:45.334079 29845 raft_consensus.cc:348] T ad79026708c64d1b863119191198722e P 436223292ce14a96b6da9826e04462b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "436223292ce14a96b6da9826e04462b2" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 35291 } } peers { permanent_uuid: "f272843f51044e3ab3fe3234eaaabbc2" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 32851 } } peers { permanent_uuid: "ceece70b4fe7465f9ae0c9f3299ffa7f" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 42023 } }
I20260810 03:25:45.334625 29845 raft_consensus.cc:374] T ad79026708c64d1b863119191198722e P 436223292ce14a96b6da9826e04462b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:25:45.334733 29845 raft_consensus.cc:729] T ad79026708c64d1b863119191198722e P 436223292ce14a96b6da9826e04462b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 436223292ce14a96b6da9826e04462b2, State: Initialized, Role: FOLLOWER
I20260810 03:25:45.335331 29843 ts_tablet_manager.cc:1428] T ad79026708c64d1b863119191198722e P f272843f51044e3ab3fe3234eaaabbc2: Time spent starting tablet: real 0.014s user 0.013s sys 0.002s
I20260810 03:25:45.335325 29845 consensus_queue.cc:260] T ad79026708c64d1b863119191198722e P 436223292ce14a96b6da9826e04462b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "436223292ce14a96b6da9826e04462b2" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 35291 } } peers { permanent_uuid: "f272843f51044e3ab3fe3234eaaabbc2" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 32851 } } peers { permanent_uuid: "ceece70b4fe7465f9ae0c9f3299ffa7f" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 42023 } }
I20260810 03:25:45.336588 29607 heartbeater.cc:503] Master 127.27.161.62:41809 was elected leader, sending a full tablet report...
I20260810 03:25:45.338002 29845 ts_tablet_manager.cc:1428] T ad79026708c64d1b863119191198722e P 436223292ce14a96b6da9826e04462b2: Time spent starting tablet: real 0.015s user 0.011s sys 0.003s
I20260810 03:25:45.338897 29852 raft_consensus.cc:482] T ad79026708c64d1b863119191198722e P 436223292ce14a96b6da9826e04462b2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 03:25:45.339208 29852 raft_consensus.cc:504] T ad79026708c64d1b863119191198722e P 436223292ce14a96b6da9826e04462b2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "436223292ce14a96b6da9826e04462b2" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 35291 } } peers { permanent_uuid: "f272843f51044e3ab3fe3234eaaabbc2" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 32851 } } peers { permanent_uuid: "ceece70b4fe7465f9ae0c9f3299ffa7f" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 42023 } }
I20260810 03:25:45.346139 29852 leader_election.cc:288] T ad79026708c64d1b863119191198722e P 436223292ce14a96b6da9826e04462b2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f272843f51044e3ab3fe3234eaaabbc2 (127.27.161.3:32851), ceece70b4fe7465f9ae0c9f3299ffa7f (127.27.161.1:42023)
I20260810 03:25:45.350135 29835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:45.342153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.161.2:53807 (local address 127.27.161.3:32851)
0810 03:25:45.342645 (+ 492us) server_negotiation.cc:206] Beginning negotiation
0810 03:25:45.342651 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:25:45.343371 (+ 720us) server_negotiation.cc:407] Connection header received
0810 03:25:45.343583 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:25:45.343588 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:25:45.343657 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:25:45.343719 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:25:45.344580 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:45.345357 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:45.348178 (+ 2821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:45.349071 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:45.349230 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:25:45.349315 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0810 03:25:45.349709 (+ 394us) server_negotiation.cc:299] Negotiation successful
0810 03:25:45.349839 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":326}
I20260810 03:25:45.350281 29854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:45.342061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.3:32851 (local address 127.27.161.2:53807)
0810 03:25:45.343265 (+ 1204us) negotiation.cc:107] Waiting for socket to connect
0810 03:25:45.343288 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0810 03:25:45.343438 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:25:45.343929 (+ 491us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:25:45.343934 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:25:45.343967 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:25:45.344404 (+ 437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:25:45.344417 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:25:45.345483 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:25:45.345489 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:25:45.348008 (+ 2519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:25:45.348025 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:25:45.349332 (+ 1307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:25:45.349340 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:25:45.349528 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:25:45.349555 (+ 27us) client_negotiation.cc:769] Sending connection context
0810 03:25:45.349663 (+ 108us) client_negotiation.cc:240] Negotiation successful
0810 03:25:45.349812 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1002,"mutex_wait_us":33,"thread_start_us":817,"threads_started":1}
I20260810 03:25:45.351557 29734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad79026708c64d1b863119191198722e" candidate_uuid: "436223292ce14a96b6da9826e04462b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f272843f51044e3ab3fe3234eaaabbc2" is_pre_election: true
I20260810 03:25:45.352330 29855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:45.343611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.1:42023 (local address 127.27.161.2:50469)
0810 03:25:45.346815 (+ 3204us) negotiation.cc:107] Waiting for socket to connect
0810 03:25:45.346832 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 03:25:45.346961 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:25:45.347269 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:25:45.347272 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:25:45.347292 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:25:45.347718 (+ 426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:25:45.347729 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:25:45.348828 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:25:45.348834 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:25:45.350385 (+ 1551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:25:45.350400 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:25:45.351637 (+ 1237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:25:45.351643 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:25:45.351803 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:25:45.351827 (+ 24us) client_negotiation.cc:769] Sending connection context
0810 03:25:45.351913 (+ 86us) client_negotiation.cc:240] Negotiation successful
0810 03:25:45.352034 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3092,"mutex_wait_us":1,"thread_start_us":2874,"threads_started":1}
I20260810 03:25:45.352380 29839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:45.343775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.161.2:50469 (local address 127.27.161.1:42023)
0810 03:25:45.343969 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0810 03:25:45.343973 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:25:45.346934 (+ 2961us) server_negotiation.cc:407] Connection header received
0810 03:25:45.347083 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:25:45.347089 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:25:45.347168 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:25:45.347262 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:25:45.347874 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:45.348687 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:45.350590 (+ 1903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:45.351428 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:45.351583 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:25:45.351670 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0810 03:25:45.351993 (+ 323us) server_negotiation.cc:299] Negotiation successful
0810 03:25:45.352139 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260810 03:25:45.352403 29734 raft_consensus.cc:2454] T ad79026708c64d1b863119191198722e P f272843f51044e3ab3fe3234eaaabbc2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 436223292ce14a96b6da9826e04462b2 in term 0.
I20260810 03:25:45.353266 29445 leader_election.cc:302] T ad79026708c64d1b863119191198722e P 436223292ce14a96b6da9826e04462b2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 436223292ce14a96b6da9826e04462b2, f272843f51044e3ab3fe3234eaaabbc2; no voters:
I20260810 03:25:45.353657 29852 raft_consensus.cc:2781] T ad79026708c64d1b863119191198722e P 436223292ce14a96b6da9826e04462b2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 03:25:45.353792 29852 raft_consensus.cc:482] T ad79026708c64d1b863119191198722e P 436223292ce14a96b6da9826e04462b2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 03:25:45.353574 29343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad79026708c64d1b863119191198722e" candidate_uuid: "436223292ce14a96b6da9826e04462b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ceece70b4fe7465f9ae0c9f3299ffa7f" is_pre_election: true
I20260810 03:25:45.353885 29852 raft_consensus.cc:3037] T ad79026708c64d1b863119191198722e P 436223292ce14a96b6da9826e04462b2 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:25:45.354199 29343 raft_consensus.cc:2454] T ad79026708c64d1b863119191198722e P ceece70b4fe7465f9ae0c9f3299ffa7f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 436223292ce14a96b6da9826e04462b2 in term 0.
I20260810 03:25:45.356607 29852 raft_consensus.cc:504] T ad79026708c64d1b863119191198722e P 436223292ce14a96b6da9826e04462b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "436223292ce14a96b6da9826e04462b2" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 35291 } } peers { permanent_uuid: "f272843f51044e3ab3fe3234eaaabbc2" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 32851 } } peers { permanent_uuid: "ceece70b4fe7465f9ae0c9f3299ffa7f" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 42023 } }
I20260810 03:25:45.357306 29852 leader_election.cc:288] T ad79026708c64d1b863119191198722e P 436223292ce14a96b6da9826e04462b2 [CANDIDATE]: Term 1 election: Requested vote from peers f272843f51044e3ab3fe3234eaaabbc2 (127.27.161.3:32851), ceece70b4fe7465f9ae0c9f3299ffa7f (127.27.161.1:42023)
I20260810 03:25:45.357717 29734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad79026708c64d1b863119191198722e" candidate_uuid: "436223292ce14a96b6da9826e04462b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f272843f51044e3ab3fe3234eaaabbc2"
I20260810 03:25:45.357777 29343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad79026708c64d1b863119191198722e" candidate_uuid: "436223292ce14a96b6da9826e04462b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ceece70b4fe7465f9ae0c9f3299ffa7f"
I20260810 03:25:45.357910 29734 raft_consensus.cc:3037] T ad79026708c64d1b863119191198722e P f272843f51044e3ab3fe3234eaaabbc2 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:25:45.358004 29343 raft_consensus.cc:3037] T ad79026708c64d1b863119191198722e P ceece70b4fe7465f9ae0c9f3299ffa7f [term 0 FOLLOWER]: Advancing to term 1
W20260810 03:25:45.359599 29405 tablet.cc:2369] T ad79026708c64d1b863119191198722e P ceece70b4fe7465f9ae0c9f3299ffa7f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 03:25:45.360491 29343 raft_consensus.cc:2454] T ad79026708c64d1b863119191198722e P ceece70b4fe7465f9ae0c9f3299ffa7f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 436223292ce14a96b6da9826e04462b2 in term 1.
I20260810 03:25:45.360950 29734 raft_consensus.cc:2454] T ad79026708c64d1b863119191198722e P f272843f51044e3ab3fe3234eaaabbc2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 436223292ce14a96b6da9826e04462b2 in term 1.
I20260810 03:25:45.360944 29442 leader_election.cc:302] T ad79026708c64d1b863119191198722e P 436223292ce14a96b6da9826e04462b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 436223292ce14a96b6da9826e04462b2, ceece70b4fe7465f9ae0c9f3299ffa7f; no voters:
I20260810 03:25:45.361456 29852 raft_consensus.cc:2781] T ad79026708c64d1b863119191198722e P 436223292ce14a96b6da9826e04462b2 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:25:45.362056 29852 raft_consensus.cc:686] T ad79026708c64d1b863119191198722e P 436223292ce14a96b6da9826e04462b2 [term 1 LEADER]: Becoming Leader. State: Replica: 436223292ce14a96b6da9826e04462b2, State: Running, Role: LEADER
I20260810 03:25:45.362555 29852 consensus_queue.cc:237] T ad79026708c64d1b863119191198722e P 436223292ce14a96b6da9826e04462b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "436223292ce14a96b6da9826e04462b2" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 35291 } } peers { permanent_uuid: "f272843f51044e3ab3fe3234eaaabbc2" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 32851 } } peers { permanent_uuid: "ceece70b4fe7465f9ae0c9f3299ffa7f" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 42023 } }
I20260810 03:25:45.368475 29115 catalog_manager.cc:5495] T ad79026708c64d1b863119191198722e P 436223292ce14a96b6da9826e04462b2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 436223292ce14a96b6da9826e04462b2 (127.27.161.2). New cstate: current_term: 1 leader_uuid: "436223292ce14a96b6da9826e04462b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "436223292ce14a96b6da9826e04462b2" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 35291 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f272843f51044e3ab3fe3234eaaabbc2" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 32851 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ceece70b4fe7465f9ae0c9f3299ffa7f" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 42023 } health_report { overall_health: UNKNOWN } } }
W20260810 03:25:45.382365 29609 tablet.cc:2369] T ad79026708c64d1b863119191198722e P 436223292ce14a96b6da9826e04462b2: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 10 03:25:45 dist-test-slave-vpvm krb5kdc[29055](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332342, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.161.1@KRBTEST.COM
I20260810 03:25:45.412029 29839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:45.393798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45024 (local address 127.27.161.1:42023)
0810 03:25:45.394093 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0810 03:25:45.394101 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 03:25:45.394650 (+ 549us) server_negotiation.cc:407] Connection header received
0810 03:25:45.394793 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:25:45.394797 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:25:45.394879 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:25:45.394965 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:25:45.396903 (+ 1938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:45.397814 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:45.399294 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:45.399830 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:45.399939 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:25:45.405230 (+ 5291us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:25:45.405254 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:25:45.405272 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:25:45.405321 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:25:45.408542 (+ 3221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:25:45.409442 (+ 900us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:25:45.409451 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:25:45.409454 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:25:45.409626 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:25:45.410165 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:25:45.410170 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:25:45.410172 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:25:45.410650 (+ 478us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:25:45.410994 (+ 344us) server_negotiation.cc:1035] Waiting for connection context
0810 03:25:45.411371 (+ 377us) server_negotiation.cc:299] Negotiation successful
0810 03:25:45.411628 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
W20260810 03:25:45.414036 29805 tablet.cc:2369] T ad79026708c64d1b863119191198722e P f272843f51044e3ab3fe3234eaaabbc2: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 10 03:25:45 dist-test-slave-vpvm krb5kdc[29055](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332345, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 10 03:25:45 dist-test-slave-vpvm krb5kdc[29055](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332345, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.27.161.1@KRBTEST.COM
I20260810 03:25:45.475905 29839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:45.460907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45034 (local address 127.27.161.1:42023)
0810 03:25:45.461186 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0810 03:25:45.461192 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:25:45.461556 (+ 364us) server_negotiation.cc:407] Connection header received
0810 03:25:45.461744 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:25:45.461749 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:25:45.461821 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:25:45.461898 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:25:45.463415 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:45.464183 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:45.465537 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:45.466069 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:45.466179 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:25:45.470244 (+ 4065us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:25:45.470250 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:25:45.470253 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:25:45.470294 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:25:45.473155 (+ 2861us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:25:45.473869 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:25:45.473874 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:25:45.473876 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:25:45.474082 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:25:45.474527 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:25:45.474531 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:25:45.474533 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:25:45.474899 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:25:45.475172 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0810 03:25:45.475396 (+ 224us) server_negotiation.cc:299] Negotiation successful
0810 03:25:45.475599 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
Aug 10 03:25:45 dist-test-slave-vpvm krb5kdc[29055](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332345, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 10 03:25:45 dist-test-slave-vpvm krb5kdc[29055](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332345, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.161.1@KRBTEST.COM
I20260810 03:25:45.537273 29839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:45.520957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45048 (local address 127.27.161.1:42023)
0810 03:25:45.521223 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0810 03:25:45.521230 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:25:45.521537 (+ 307us) server_negotiation.cc:407] Connection header received
0810 03:25:45.521743 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:25:45.521748 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:25:45.521817 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:25:45.521900 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:25:45.523548 (+ 1648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:45.524256 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:45.525607 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:45.526178 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:45.526302 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:25:45.530698 (+ 4396us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:25:45.530704 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:25:45.530707 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:25:45.530802 (+ 95us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:25:45.533681 (+ 2879us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:25:45.534645 (+ 964us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:25:45.534650 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:25:45.534652 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:25:45.534854 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:25:45.535317 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:25:45.535322 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:25:45.535324 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:25:45.535889 (+ 565us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:25:45.536154 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0810 03:25:45.536539 (+ 385us) server_negotiation.cc:299] Negotiation successful
0810 03:25:45.536809 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
W20260810 03:25:45.539922 29282 tablet_service.cc:2908] Rejecting scan request for tablet ad79026708c64d1b863119191198722e: Uninitialized: clean time has not yet been initialized
I20260810 03:25:45.544169 28292 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iBH5/build/asan/bin/kudu pid 29218
I20260810 03:25:45.557650 29839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:45.545140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45060 (local address 127.27.161.1:42023)
0810 03:25:45.545316 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0810 03:25:45.545320 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:25:45.545654 (+ 334us) server_negotiation.cc:407] Connection header received
0810 03:25:45.545882 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:25:45.545888 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:25:45.546037 (+ 149us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:25:45.546144 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:25:45.547512 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:45.548286 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:45.549511 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:45.549971 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:45.550101 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:25:45.552357 (+ 2256us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:25:45.552362 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:25:45.552365 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:25:45.552402 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:25:45.555046 (+ 2644us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:25:45.555805 (+ 759us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:25:45.555810 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:25:45.555812 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:25:45.555948 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:25:45.556363 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:25:45.556366 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:25:45.556368 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:25:45.556674 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:25:45.556895 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0810 03:25:45.557191 (+ 296us) server_negotiation.cc:299] Negotiation successful
0810 03:25:45.557376 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260810 03:25:45.558385 29396 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:25:45.649943 28292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iBH5/build/asan/bin/kudu with pid 29218
I20260810 03:25:45.675155 28292 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iBH5/build/asan/bin/kudu pid 29418
Aug 10 03:25:45 dist-test-slave-vpvm krb5kdc[29055](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332345, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.161.2@KRBTEST.COM
I20260810 03:25:45.690969 29831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:45.675995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43192 (local address 127.27.161.2:35291)
0810 03:25:45.676210 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0810 03:25:45.676216 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:25:45.676238 (+ 22us) server_negotiation.cc:407] Connection header received
0810 03:25:45.676345 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:25:45.676349 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:25:45.676415 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:25:45.676504 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:25:45.677734 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:45.678583 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:45.679926 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:45.680367 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:45.680485 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:25:45.684766 (+ 4281us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:25:45.684785 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:25:45.684803 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:25:45.684850 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:25:45.687849 (+ 2999us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:25:45.688638 (+ 789us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:25:45.688647 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:25:45.688650 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:25:45.688837 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:25:45.689239 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:25:45.689243 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:25:45.689246 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:25:45.689660 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:25:45.690055 (+ 395us) server_negotiation.cc:1035] Waiting for connection context
0810 03:25:45.690341 (+ 286us) server_negotiation.cc:299] Negotiation successful
0810 03:25:45.690626 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260810 03:25:45.691709 29602 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:25:45.783185 28292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iBH5/build/asan/bin/kudu with pid 29418
I20260810 03:25:45.809355 28292 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iBH5/build/asan/bin/kudu pid 29619
Aug 10 03:25:45 dist-test-slave-vpvm krb5kdc[29055](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332345, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.161.3@KRBTEST.COM
I20260810 03:25:45.823542 29835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:45.810106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55678 (local address 127.27.161.3:32851)
0810 03:25:45.810307 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0810 03:25:45.810312 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:25:45.810332 (+ 20us) server_negotiation.cc:407] Connection header received
0810 03:25:45.810479 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:25:45.810483 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:25:45.810565 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:25:45.810650 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:25:45.811834 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:45.812559 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:45.813787 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:45.814308 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:45.814401 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:25:45.818324 (+ 3923us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:25:45.818340 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:25:45.818358 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:25:45.818397 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:25:45.820949 (+ 2552us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:25:45.821618 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:25:45.821626 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:25:45.821629 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:25:45.821769 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:25:45.822164 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:25:45.822168 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:25:45.822170 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:25:45.822539 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:25:45.822762 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0810 03:25:45.822998 (+ 236us) server_negotiation.cc:299] Negotiation successful
0810 03:25:45.823209 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260810 03:25:45.824174 29796 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:25:45.912307 28292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iBH5/build/asan/bin/kudu with pid 29619
I20260810 03:25:45.936383 28292 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iBH5/build/asan/bin/kudu pid 29070
I20260810 03:25:45.937834 29166 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:25:46.048249 28292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iBH5/build/asan/bin/kudu with pid 29070
2026-08-10T03:25:46Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (4501 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786332336319544-28292-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 03:25:46 dist-test-slave-vpvm krb5kdc[29885](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 03:25:46 dist-test-slave-vpvm krb5kdc[29885](info): set up 2 sockets
Aug 10 03:25:46 dist-test-slave-vpvm krb5kdc[29885](info): commencing operation
krb5kdc: starting...
W20260810 03:25:47.257095 28292 mini_kdc.cc:121] Time spent starting KDC: real 1.158s user 0.001s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786332336319544-28292-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786332336319544-28292-0/krb5kdc/test-user.keytab.
Aug 10 03:25:47 dist-test-slave-vpvm krb5kdc[29885](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332347, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T03:25:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T03:25:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.161.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.161.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.161.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.62.keytab.
Entry for principal kudu/127.27.161.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.62.keytab.
I20260810 03:25:47.483918 28292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iBH5/build/asan/bin/kudu
/tmp/dist-test-task24iBH5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786332336319544-28292-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786332336319544-28292-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786332336319544-28292-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786332336319544-28292-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.161.62:38165
--webserver_interface=127.27.161.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.62.keytab
--principal=kudu/127.27.161.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.161.20:46169
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.161.62:38165
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786332336319544-28292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786332336319544-28292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786332336319544-28292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:25:47.827014 29902 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:25:47.827335 29902 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:25:47.827459 29902 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:25:47.835358 29902 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:25:47.835484 29902 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260810 03:25:47.835525 29902 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:25:47.835611 29902 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:25:47.835673 29902 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:25:47.835759 29902 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:25:47.846203 29902 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.161.20:46169
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786332336319544-28292-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786332336319544-28292-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.161.62:38165
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.62.keytab
--principal=kudu/127.27.161.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.161.62:38165
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786332336319544-28292-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.161.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786332336319544-28292-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:25:47.847651 29902 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:25:47.849157 29902 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:25:47.861464 29913 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:25:47.861543 29912 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:25:47.861583 29917 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:25:47.862566 29902 server_base.cc:1034] running on GCE node
I20260810 03:25:47.863570 29902 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:25:47.865278 29902 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:25:47.866561 29902 hybrid_clock.cc:648] HybridClock initialized: now 1786332347866472 us; error 54 us; skew 500 ppm
I20260810 03:25:47.866961 29902 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:25:47 dist-test-slave-vpvm krb5kdc[29885](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332347, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.161.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:25:47.870381 29902 init.cc:377] Logged in from keytab as kudu/127.27.161.62@KRBTEST.COM (short username kudu)
I20260810 03:25:47.872037 29902 webserver.cc:460] Webserver started at http://127.27.161.62:37317/ using document root <none> and password file <none>
I20260810 03:25:47.872820 29902 fs_manager.cc:359] Metadata directory not provided
I20260810 03:25:47.872934 29902 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:25:47.873270 29902 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:25:47.876554 29902 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786332336319544-28292-0/minicluster-data/master-0/data/instance:
uuid: "58a44e566ba2481daf3c673adfce18c7"
format_stamp: "Formatted at 2026-08-10 03:25:47 on dist-test-slave-vpvm"
server_key: "d26bce0fa0e4474179a0c36c611a2593"
server_key_iv: "72df7a7590927b273676cb8d8ac05da4"
server_key_version: "encryptionkey@0"
I20260810 03:25:47.877467 29902 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786332336319544-28292-0/minicluster-data/master-0/wal/instance:
uuid: "58a44e566ba2481daf3c673adfce18c7"
format_stamp: "Formatted at 2026-08-10 03:25:47 on dist-test-slave-vpvm"
server_key: "d26bce0fa0e4474179a0c36c611a2593"
server_key_iv: "72df7a7590927b273676cb8d8ac05da4"
server_key_version: "encryptionkey@0"
I20260810 03:25:47.883195 29902 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260810 03:25:47.887223 29929 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:25:47.888698 29902 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260810 03:25:47.888892 29902 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786332336319544-28292-0/minicluster-data/master-0/data,/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786332336319544-28292-0/minicluster-data/master-0/wal
uuid: "58a44e566ba2481daf3c673adfce18c7"
format_stamp: "Formatted at 2026-08-10 03:25:47 on dist-test-slave-vpvm"
server_key: "d26bce0fa0e4474179a0c36c611a2593"
server_key_iv: "72df7a7590927b273676cb8d8ac05da4"
server_key_version: "encryptionkey@0"
I20260810 03:25:47.889138 29902 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786332336319544-28292-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786332336319544-28292-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786332336319544-28292-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:25:47.925254 29902 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:25:47.930187 29902 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:25:47.930471 29902 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:25:47.952325 29902 rpc_server.cc:307] RPC server started. Bound to: 127.27.161.62:38165
I20260810 03:25:47.952351 30020 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.161.62:38165 every 8 connection(s)
I20260810 03:25:47.953785 29902 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786332336319544-28292-0/minicluster-data/master-0/data/info.pb
I20260810 03:25:47.957321 28292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iBH5/build/asan/bin/kudu as pid 29902
I20260810 03:25:47.957597 28292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786332336319544-28292-0/minicluster-data/master-0/wal/instance
I20260810 03:25:47.957930 30021 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:25:47.958323 28292 external_mini_cluster.cc:1428] Setting key f841e4258ace6d6b538ae9464b300fb9
I20260810 03:25:47.973445 30021 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 58a44e566ba2481daf3c673adfce18c7: Bootstrap starting.
Aug 10 03:25:47 dist-test-slave-vpvm krb5kdc[29885](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332347, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.161.62@KRBTEST.COM
I20260810 03:25:47.977598 30021 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 58a44e566ba2481daf3c673adfce18c7: Neither blocks nor log segments found. Creating new log.
I20260810 03:25:47.979158 30021 log.cc:824] T 00000000000000000000000000000000 P 58a44e566ba2481daf3c673adfce18c7: Log is configured to *not* fsync() on all Append() calls
I20260810 03:25:47.983294 30021 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 58a44e566ba2481daf3c673adfce18c7: No bootstrap required, opened a new log
I20260810 03:25:47.985410 30025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:47.960587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32828 (local address 127.27.161.62:38165)
0810 03:25:47.961560 (+ 973us) server_negotiation.cc:206] Beginning negotiation
0810 03:25:47.961576 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 03:25:47.961637 (+ 61us) server_negotiation.cc:407] Connection header received
0810 03:25:47.963549 (+ 1912us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:25:47.963592 (+ 43us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:25:47.964260 (+ 668us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:25:47.964787 (+ 527us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:25:47.966248 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:47.968220 (+ 1972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:47.969632 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:47.970852 (+ 1220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:47.971065 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:25:47.975352 (+ 4287us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:25:47.975370 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:25:47.975393 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:25:47.975460 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:25:47.979440 (+ 3980us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:25:47.980480 (+ 1040us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:25:47.980489 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:25:47.980497 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:25:47.980704 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:25:47.981481 (+ 777us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:25:47.981487 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:25:47.981491 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:25:47.982335 (+ 844us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:25:47.982742 (+ 407us) server_negotiation.cc:1035] Waiting for connection context
0810 03:25:47.983244 (+ 502us) server_negotiation.cc:299] Negotiation successful
0810 03:25:47.983929 (+ 685us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":671,"thread_start_us":406,"threads_started":1}
I20260810 03:25:47.992349 30021 raft_consensus.cc:348] T 00000000000000000000000000000000 P 58a44e566ba2481daf3c673adfce18c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58a44e566ba2481daf3c673adfce18c7" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 38165 } }
I20260810 03:25:47.992830 30021 raft_consensus.cc:374] T 00000000000000000000000000000000 P 58a44e566ba2481daf3c673adfce18c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:25:47.992939 30021 raft_consensus.cc:729] T 00000000000000000000000000000000 P 58a44e566ba2481daf3c673adfce18c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58a44e566ba2481daf3c673adfce18c7, State: Initialized, Role: FOLLOWER
I20260810 03:25:47.993505 30021 consensus_queue.cc:260] T 00000000000000000000000000000000 P 58a44e566ba2481daf3c673adfce18c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58a44e566ba2481daf3c673adfce18c7" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 38165 } }
I20260810 03:25:47.993773 30021 raft_consensus.cc:388] T 00000000000000000000000000000000 P 58a44e566ba2481daf3c673adfce18c7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 03:25:47.993906 30021 raft_consensus.cc:482] T 00000000000000000000000000000000 P 58a44e566ba2481daf3c673adfce18c7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 03:25:47.994083 30021 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 58a44e566ba2481daf3c673adfce18c7 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:25:47.996333 30021 raft_consensus.cc:504] T 00000000000000000000000000000000 P 58a44e566ba2481daf3c673adfce18c7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58a44e566ba2481daf3c673adfce18c7" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 38165 } }
I20260810 03:25:47.996857 30021 leader_election.cc:302] T 00000000000000000000000000000000 P 58a44e566ba2481daf3c673adfce18c7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 58a44e566ba2481daf3c673adfce18c7; no voters:
I20260810 03:25:47.997400 30021 leader_election.cc:288] T 00000000000000000000000000000000 P 58a44e566ba2481daf3c673adfce18c7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 03:25:47.997568 30027 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 58a44e566ba2481daf3c673adfce18c7 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:25:47.998589 30027 raft_consensus.cc:686] T 00000000000000000000000000000000 P 58a44e566ba2481daf3c673adfce18c7 [term 1 LEADER]: Becoming Leader. State: Replica: 58a44e566ba2481daf3c673adfce18c7, State: Running, Role: LEADER
I20260810 03:25:47.999195 30027 consensus_queue.cc:237] T 00000000000000000000000000000000 P 58a44e566ba2481daf3c673adfce18c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58a44e566ba2481daf3c673adfce18c7" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 38165 } }
I20260810 03:25:47.999758 30021 sys_catalog.cc:564] T 00000000000000000000000000000000 P 58a44e566ba2481daf3c673adfce18c7 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:25:48.002636 30029 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58a44e566ba2481daf3c673adfce18c7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 58a44e566ba2481daf3c673adfce18c7. Latest consensus state: current_term: 1 leader_uuid: "58a44e566ba2481daf3c673adfce18c7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58a44e566ba2481daf3c673adfce18c7" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 38165 } } }
I20260810 03:25:48.002592 30028 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58a44e566ba2481daf3c673adfce18c7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "58a44e566ba2481daf3c673adfce18c7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58a44e566ba2481daf3c673adfce18c7" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 38165 } } }
I20260810 03:25:48.003069 30028 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58a44e566ba2481daf3c673adfce18c7 [sys.catalog]: This master's current role is: LEADER
I20260810 03:25:48.003073 30029 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58a44e566ba2481daf3c673adfce18c7 [sys.catalog]: This master's current role is: LEADER
I20260810 03:25:48.005208 30040 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 03:25:48.010573 30040 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 03:25:48.020314 30040 catalog_manager.cc:1347] Generated new cluster ID: 61084cb888824ee58e2f2688a178d0ea
I20260810 03:25:48.020466 30040 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 03:25:48.043071 30040 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 03:25:48.044842 30040 catalog_manager.cc:1504] Loading token signing keys...
I20260810 03:25:48.053864 30040 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 58a44e566ba2481daf3c673adfce18c7: Generated new TSK 0
I20260810 03:25:48.055095 30040 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.161.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.161.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.161.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.1.keytab.
Entry for principal kudu/127.27.161.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.1.keytab.
I20260810 03:25:48.117128 28292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iBH5/build/asan/bin/kudu
/tmp/dist-test-task24iBH5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786332336319544-28292-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786332336319544-28292-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786332336319544-28292-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786332336319544-28292-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.161.1:0
--local_ip_for_outbound_sockets=127.27.161.1
--webserver_interface=127.27.161.1
--webserver_port=0
--tserver_master_addrs=127.27.161.62:38165
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.1.keytab
--principal=kudu/127.27.161.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.161.20:46169
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786332336319544-28292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786332336319544-28292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786332336319544-28292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:25:48.472267 30061 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:25:48.472579 30061 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:25:48.472707 30061 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:25:48.480315 30061 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:25:48.480465 30061 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:25:48.480599 30061 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 03:25:48.480715 30061 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.161.1
I20260810 03:25:48.491647 30061 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.161.20:46169
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786332336319544-28292-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786332336319544-28292-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.1.keytab
--principal=kudu/127.27.161.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.161.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786332336319544-28292-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.161.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.27.161.62:38165
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.161.1
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786332336319544-28292-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:25:48.493100 30061 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:25:48.494720 30061 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:25:48.509267 30075 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:25:48.509341 30069 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:25:48.509372 30070 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:25:48.510291 30061 server_base.cc:1034] running on GCE node
I20260810 03:25:48.510958 30061 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:25:48.512491 30061 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:25:48.513756 30061 hybrid_clock.cc:648] HybridClock initialized: now 1786332348513657 us; error 58 us; skew 500 ppm
I20260810 03:25:48.514145 30061 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:25:48 dist-test-slave-vpvm krb5kdc[29885](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332348, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.161.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:25:48.517845 30061 init.cc:377] Logged in from keytab as kudu/127.27.161.1@KRBTEST.COM (short username kudu)
I20260810 03:25:48.519724 30061 webserver.cc:460] Webserver started at http://127.27.161.1:42701/ using document root <none> and password file <none>
I20260810 03:25:48.520509 30061 fs_manager.cc:359] Metadata directory not provided
I20260810 03:25:48.520628 30061 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:25:48.520969 30061 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:25:48.524334 30061 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786332336319544-28292-0/minicluster-data/ts-0/data/instance:
uuid: "74e398a49767421291bb348c467b01c2"
format_stamp: "Formatted at 2026-08-10 03:25:48 on dist-test-slave-vpvm"
server_key: "f1ac675fad448d9d25386298a522de10"
server_key_iv: "ea29f5dee9317797eed4b0b5ba5b7dab"
server_key_version: "encryptionkey@0"
I20260810 03:25:48.525203 30061 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786332336319544-28292-0/minicluster-data/ts-0/wal/instance:
uuid: "74e398a49767421291bb348c467b01c2"
format_stamp: "Formatted at 2026-08-10 03:25:48 on dist-test-slave-vpvm"
server_key: "f1ac675fad448d9d25386298a522de10"
server_key_iv: "ea29f5dee9317797eed4b0b5ba5b7dab"
server_key_version: "encryptionkey@0"
I20260810 03:25:48.530802 30061 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260810 03:25:48.534667 30087 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:25:48.536193 30061 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 03:25:48.536381 30061 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786332336319544-28292-0/minicluster-data/ts-0/data,/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786332336319544-28292-0/minicluster-data/ts-0/wal
uuid: "74e398a49767421291bb348c467b01c2"
format_stamp: "Formatted at 2026-08-10 03:25:48 on dist-test-slave-vpvm"
server_key: "f1ac675fad448d9d25386298a522de10"
server_key_iv: "ea29f5dee9317797eed4b0b5ba5b7dab"
server_key_version: "encryptionkey@0"
I20260810 03:25:48.536653 30061 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786332336319544-28292-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786332336319544-28292-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786332336319544-28292-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:25:48.582674 30061 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:25:48.587105 30061 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:25:48.587405 30061 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:25:48.590525 30061 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:25:48.590622 30061 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:25:48.590768 30061 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:25:48.590827 30061 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 03:25:48 dist-test-slave-vpvm krb5kdc[29885](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332348, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.161.1@KRBTEST.COM for kudu/127.27.161.62@KRBTEST.COM
I20260810 03:25:48.615576 30099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:48.591351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.161.1:40021 (local address 127.27.161.62:38165)
0810 03:25:48.591864 (+ 513us) server_negotiation.cc:206] Beginning negotiation
0810 03:25:48.591871 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:25:48.593299 (+ 1428us) server_negotiation.cc:407] Connection header received
0810 03:25:48.595842 (+ 2543us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:25:48.595850 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:25:48.595951 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:25:48.596043 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:25:48.598945 (+ 2902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:48.600129 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:48.602862 (+ 2733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:48.603386 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:48.603528 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:25:48.608289 (+ 4761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:25:48.608295 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:25:48.608299 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:25:48.608338 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:25:48.611598 (+ 3260us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:25:48.612796 (+ 1198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:25:48.612801 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:25:48.612804 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:25:48.612947 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:25:48.613562 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:25:48.613567 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:25:48.613569 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:25:48.613884 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:25:48.614073 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0810 03:25:48.615104 (+ 1031us) server_negotiation.cc:299] Negotiation successful
0810 03:25:48.615313 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":376,"thread_start_us":265,"threads_started":1}
I20260810 03:25:48.617643 30102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:48.592082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:38165 (local address 127.27.161.1:40021)
0810 03:25:48.593085 (+ 1003us) negotiation.cc:107] Waiting for socket to connect
0810 03:25:48.593150 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0810 03:25:48.595426 (+ 2276us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:25:48.596650 (+ 1224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:25:48.596663 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:25:48.597610 (+ 947us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:25:48.598651 (+ 1041us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:25:48.598681 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:25:48.600376 (+ 1695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:25:48.600382 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:25:48.602648 (+ 2266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:25:48.602687 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:25:48.603666 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:25:48.603676 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:25:48.603910 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:25:48.604598 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:25:48.604622 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:25:48.607935 (+ 3313us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:25:48.611801 (+ 3866us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:25:48.611815 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:25:48.611839 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:25:48.612614 (+ 775us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:25:48.613060 (+ 446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:25:48.613066 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:25:48.613070 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:25:48.613392 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:25:48.614044 (+ 652us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:25:48.614055 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:25:48.614642 (+ 587us) client_negotiation.cc:769] Sending connection context
0810 03:25:48.614998 (+ 356us) client_negotiation.cc:240] Negotiation successful
0810 03:25:48.615697 (+ 699us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":699,"thread_start_us":413,"threads_started":1}
I20260810 03:25:48.642119 30200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:48.625281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:38165 (local address 127.27.161.1:47721)
0810 03:25:48.626255 (+ 974us) negotiation.cc:107] Waiting for socket to connect
0810 03:25:48.626294 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0810 03:25:48.626612 (+ 318us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:25:48.627288 (+ 676us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:25:48.627295 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:25:48.628069 (+ 774us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:25:48.628541 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:25:48.628564 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:25:48.629675 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:25:48.629682 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:25:48.631457 (+ 1775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:25:48.631481 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:25:48.632565 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:25:48.632572 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:25:48.632725 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:25:48.633447 (+ 722us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:25:48.633473 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:25:48.635000 (+ 1527us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:25:48.638463 (+ 3463us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:25:48.638469 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:25:48.638473 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:25:48.639120 (+ 647us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:25:48.639603 (+ 483us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:25:48.639612 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:25:48.639616 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:25:48.639766 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:25:48.640714 (+ 948us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:25:48.640720 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:25:48.640940 (+ 220us) client_negotiation.cc:769] Sending connection context
0810 03:25:48.641113 (+ 173us) client_negotiation.cc:240] Negotiation successful
0810 03:25:48.641523 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":605,"thread_start_us":421,"threads_started":1}
I20260810 03:25:48.642808 30099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:48.625350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.161.1:47721 (local address 127.27.161.62:38165)
0810 03:25:48.625576 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0810 03:25:48.625584 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 03:25:48.626482 (+ 898us) server_negotiation.cc:407] Connection header received
0810 03:25:48.626809 (+ 327us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:25:48.626816 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:25:48.626924 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:25:48.627093 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:25:48.628773 (+ 1680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:48.629515 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:48.631717 (+ 2202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:48.632338 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:48.632467 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:25:48.635271 (+ 2804us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:25:48.635279 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:25:48.635284 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:25:48.635332 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:25:48.638296 (+ 2964us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:25:48.639296 (+ 1000us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:25:48.639302 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:25:48.639306 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:25:48.639488 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:25:48.639943 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:25:48.639949 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:25:48.639952 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:25:48.640315 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:25:48.640478 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0810 03:25:48.642192 (+ 1714us) server_negotiation.cc:299] Negotiation successful
0810 03:25:48.642479 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260810 03:25:48.649281 30061 rpc_server.cc:307] RPC server started. Bound to: 127.27.161.1:34289
I20260810 03:25:48.649313 30276 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.161.1:34289 every 8 connection(s)
I20260810 03:25:48.650570 30061 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786332336319544-28292-0/minicluster-data/ts-0/data/info.pb
I20260810 03:25:48.652694 30278 heartbeater.cc:347] Connected to a master server at 127.27.161.62:38165
I20260810 03:25:48.652916 30278 heartbeater.cc:464] Registering TS with master...
I20260810 03:25:48.653577 30278 heartbeater.cc:511] Master 127.27.161.62:38165 requested a full tablet report, sending...
I20260810 03:25:48.654330 28292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iBH5/build/asan/bin/kudu as pid 30061
I20260810 03:25:48.654556 28292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786332336319544-28292-0/minicluster-data/ts-0/wal/instance
I20260810 03:25:48.655207 28292 external_mini_cluster.cc:1428] Setting key db864d75876ea7b70f1248b28f08f43a
I20260810 03:25:48.656060 29951 ts_manager.cc:194] Registered new tserver with Master: 74e398a49767421291bb348c467b01c2 (127.27.161.1:34289)
I20260810 03:25:48.657793 29951 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.161.1@KRBTEST.COM'} at 127.27.161.1:40021
WARNING: no policy specified for kudu/127.27.161.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.161.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.161.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.2.keytab.
Entry for principal kudu/127.27.161.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.2.keytab.
I20260810 03:25:48.703418 28292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iBH5/build/asan/bin/kudu
/tmp/dist-test-task24iBH5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786332336319544-28292-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786332336319544-28292-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786332336319544-28292-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786332336319544-28292-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.161.2:0
--local_ip_for_outbound_sockets=127.27.161.2
--webserver_interface=127.27.161.2
--webserver_port=0
--tserver_master_addrs=127.27.161.62:38165
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.2.keytab
--principal=kudu/127.27.161.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.161.20:46169
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786332336319544-28292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786332336319544-28292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786332336319544-28292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:25:49.054085 30282 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:25:49.054428 30282 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:25:49.054582 30282 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:25:49.062637 30282 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:25:49.062798 30282 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:25:49.062922 30282 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 03:25:49.063033 30282 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.161.2
I20260810 03:25:49.073803 30282 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.161.20:46169
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786332336319544-28292-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786332336319544-28292-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.2.keytab
--principal=kudu/127.27.161.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.161.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786332336319544-28292-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.161.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.27.161.62:38165
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.161.2
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786332336319544-28292-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:25:49.075237 30282 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:25:49.076665 30282 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:25:49.090268 30288 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:25:49.090354 30291 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:25:49.090313 30289 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:25:49.091243 30282 server_base.cc:1034] running on GCE node
I20260810 03:25:49.091995 30282 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:25:49.093158 30282 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:25:49.094399 30282 hybrid_clock.cc:648] HybridClock initialized: now 1786332349094330 us; error 45 us; skew 500 ppm
I20260810 03:25:49.094729 30282 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:25:49 dist-test-slave-vpvm krb5kdc[29885](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332349, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.161.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:25:49.102181 30282 init.cc:377] Logged in from keytab as kudu/127.27.161.2@KRBTEST.COM (short username kudu)
I20260810 03:25:49.104022 30282 webserver.cc:460] Webserver started at http://127.27.161.2:38959/ using document root <none> and password file <none>
I20260810 03:25:49.104835 30282 fs_manager.cc:359] Metadata directory not provided
I20260810 03:25:49.104952 30282 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:25:49.105332 30282 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:25:49.108611 30282 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786332336319544-28292-0/minicluster-data/ts-1/data/instance:
uuid: "0d3f86f8815e4e28a37d1c6d9385da9e"
format_stamp: "Formatted at 2026-08-10 03:25:49 on dist-test-slave-vpvm"
server_key: "a2d45d0adf8d2e52ba04c0308d9d763e"
server_key_iv: "da6a7493245cd07b321726eb7371aa4c"
server_key_version: "encryptionkey@0"
I20260810 03:25:49.109480 30282 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786332336319544-28292-0/minicluster-data/ts-1/wal/instance:
uuid: "0d3f86f8815e4e28a37d1c6d9385da9e"
format_stamp: "Formatted at 2026-08-10 03:25:49 on dist-test-slave-vpvm"
server_key: "a2d45d0adf8d2e52ba04c0308d9d763e"
server_key_iv: "da6a7493245cd07b321726eb7371aa4c"
server_key_version: "encryptionkey@0"
I20260810 03:25:49.115459 30282 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260810 03:25:49.119159 30310 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:25:49.120581 30282 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 03:25:49.120743 30282 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786332336319544-28292-0/minicluster-data/ts-1/data,/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786332336319544-28292-0/minicluster-data/ts-1/wal
uuid: "0d3f86f8815e4e28a37d1c6d9385da9e"
format_stamp: "Formatted at 2026-08-10 03:25:49 on dist-test-slave-vpvm"
server_key: "a2d45d0adf8d2e52ba04c0308d9d763e"
server_key_iv: "da6a7493245cd07b321726eb7371aa4c"
server_key_version: "encryptionkey@0"
I20260810 03:25:49.120954 30282 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786332336319544-28292-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786332336319544-28292-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786332336319544-28292-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:25:49.139935 30282 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:25:49.144632 30282 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:25:49.144917 30282 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:25:49.147877 30282 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:25:49.147990 30282 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:25:49.148188 30282 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:25:49.148257 30282 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 03:25:49 dist-test-slave-vpvm krb5kdc[29885](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332349, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.161.2@KRBTEST.COM for kudu/127.27.161.62@KRBTEST.COM
I20260810 03:25:49.177211 30322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:49.148522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.161.2:37151 (local address 127.27.161.62:38165)
0810 03:25:49.149173 (+ 651us) server_negotiation.cc:206] Beginning negotiation
0810 03:25:49.149181 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 03:25:49.151552 (+ 2371us) server_negotiation.cc:407] Connection header received
0810 03:25:49.155026 (+ 3474us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:25:49.155034 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:25:49.155143 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:25:49.155285 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:25:49.157495 (+ 2210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:49.158607 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:49.161475 (+ 2868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:49.162130 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:49.162276 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:25:49.168300 (+ 6024us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:25:49.168307 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:25:49.168310 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:25:49.168351 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:25:49.171647 (+ 3296us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:25:49.172665 (+ 1018us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:25:49.172672 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:25:49.172676 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:25:49.172824 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:25:49.173455 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:25:49.173461 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:25:49.173465 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:25:49.173776 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:25:49.173935 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0810 03:25:49.176595 (+ 2660us) server_negotiation.cc:299] Negotiation successful
0810 03:25:49.176831 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":517,"thread_start_us":403,"threads_started":1}
I20260810 03:25:49.178689 30323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:49.149393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:38165 (local address 127.27.161.2:37151)
0810 03:25:49.151339 (+ 1946us) negotiation.cc:107] Waiting for socket to connect
0810 03:25:49.151397 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0810 03:25:49.154624 (+ 3227us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:25:49.155570 (+ 946us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:25:49.155583 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:25:49.156362 (+ 779us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:25:49.157191 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:25:49.157216 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:25:49.158820 (+ 1604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:25:49.158830 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:25:49.161281 (+ 2451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:25:49.161300 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:25:49.162651 (+ 1351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:25:49.162663 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:25:49.163063 (+ 400us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:25:49.164025 (+ 962us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:25:49.164076 (+ 51us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:25:49.167975 (+ 3899us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:25:49.171818 (+ 3843us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:25:49.171832 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:25:49.171849 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:25:49.172521 (+ 672us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:25:49.172979 (+ 458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:25:49.172994 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:25:49.172998 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:25:49.173327 (+ 329us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:25:49.173935 (+ 608us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:25:49.173945 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:25:49.174547 (+ 602us) client_negotiation.cc:769] Sending connection context
0810 03:25:49.174946 (+ 399us) client_negotiation.cc:240] Negotiation successful
0810 03:25:49.175647 (+ 701us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1595,"thread_start_us":423,"threads_started":1}
I20260810 03:25:49.196549 30282 rpc_server.cc:307] RPC server started. Bound to: 127.27.161.2:44877
I20260810 03:25:49.196631 30485 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.161.2:44877 every 8 connection(s)
I20260810 03:25:49.198093 30282 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786332336319544-28292-0/minicluster-data/ts-1/data/info.pb
I20260810 03:25:49.200582 30486 heartbeater.cc:347] Connected to a master server at 127.27.161.62:38165
I20260810 03:25:49.200866 30486 heartbeater.cc:464] Registering TS with master...
I20260810 03:25:49.201783 30486 heartbeater.cc:511] Master 127.27.161.62:38165 requested a full tablet report, sending...
I20260810 03:25:49.203377 30322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:49.187626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.161.2:50971 (local address 127.27.161.62:38165)
0810 03:25:49.187792 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0810 03:25:49.187796 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:25:49.188561 (+ 765us) server_negotiation.cc:407] Connection header received
0810 03:25:49.188838 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:25:49.188843 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:25:49.188912 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:25:49.188996 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:25:49.190524 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:49.191480 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:49.193567 (+ 2087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:49.194277 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:49.194395 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:25:49.197061 (+ 2666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:25:49.197069 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:25:49.197075 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:25:49.197120 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:25:49.200201 (+ 3081us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:25:49.201097 (+ 896us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:25:49.201102 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:25:49.201104 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:25:49.201242 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:25:49.201682 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:25:49.201687 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:25:49.201689 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:25:49.202134 (+ 445us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:25:49.202273 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0810 03:25:49.202819 (+ 546us) server_negotiation.cc:299] Negotiation successful
0810 03:25:49.203069 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260810 03:25:49.203405 30453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:49.187713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:38165 (local address 127.27.161.2:50971)
0810 03:25:49.188428 (+ 715us) negotiation.cc:107] Waiting for socket to connect
0810 03:25:49.188449 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0810 03:25:49.188695 (+ 246us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:25:49.189061 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:25:49.189078 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:25:49.189776 (+ 698us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:25:49.190353 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:25:49.190374 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:25:49.191677 (+ 1303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:25:49.191681 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:25:49.193409 (+ 1728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:25:49.193428 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:25:49.194595 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:25:49.194607 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:25:49.194782 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:25:49.195599 (+ 817us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:25:49.195623 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:25:49.196813 (+ 1190us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:25:49.200352 (+ 3539us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:25:49.200358 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:25:49.200363 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:25:49.200976 (+ 613us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:25:49.201364 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:25:49.201369 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:25:49.201372 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:25:49.201551 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:25:49.202261 (+ 710us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:25:49.202267 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:25:49.202498 (+ 231us) client_negotiation.cc:769] Sending connection context
0810 03:25:49.202670 (+ 172us) client_negotiation.cc:240] Negotiation successful
0810 03:25:49.202994 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":527,"thread_start_us":366,"threads_started":1}
I20260810 03:25:49.203642 29951 ts_manager.cc:194] Registered new tserver with Master: 0d3f86f8815e4e28a37d1c6d9385da9e (127.27.161.2:44877)
I20260810 03:25:49.204569 29951 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.161.2@KRBTEST.COM'} at 127.27.161.2:37151
I20260810 03:25:49.207489 28292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iBH5/build/asan/bin/kudu as pid 30282
I20260810 03:25:49.207680 28292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786332336319544-28292-0/minicluster-data/ts-1/wal/instance
I20260810 03:25:49.208096 28292 external_mini_cluster.cc:1428] Setting key 88fe7720f5a70478902eea1aa7b75c14
WARNING: no policy specified for kudu/127.27.161.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.161.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.161.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.3.keytab.
Entry for principal kudu/127.27.161.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.3.keytab.
I20260810 03:25:49.257582 28292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iBH5/build/asan/bin/kudu
/tmp/dist-test-task24iBH5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786332336319544-28292-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786332336319544-28292-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786332336319544-28292-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786332336319544-28292-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.161.3:0
--local_ip_for_outbound_sockets=127.27.161.3
--webserver_interface=127.27.161.3
--webserver_port=0
--tserver_master_addrs=127.27.161.62:38165
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.3.keytab
--principal=kudu/127.27.161.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.161.20:46169
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786332336319544-28292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786332336319544-28292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786332336319544-28292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:25:49.605393 30492 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:25:49.605724 30492 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:25:49.605875 30492 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:25:49.613457 30492 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:25:49.613605 30492 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:25:49.613699 30492 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 03:25:49.613782 30492 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.161.3
I20260810 03:25:49.624017 30492 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.161.20:46169
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786332336319544-28292-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786332336319544-28292-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.3.keytab
--principal=kudu/127.27.161.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.161.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786332336319544-28292-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.161.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.27.161.62:38165
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.161.3
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786332336319544-28292-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:25:49.625389 30492 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:25:49.626819 30492 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:25:49.640419 30497 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:25:49.640393 30498 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:25:49.640470 30502 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:25:49.641247 30492 server_base.cc:1034] running on GCE node
I20260810 03:25:49.641929 30492 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:25:49.642992 30492 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:25:49.644227 30492 hybrid_clock.cc:648] HybridClock initialized: now 1786332349644153 us; error 47 us; skew 500 ppm
I20260810 03:25:49.644539 30492 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:25:49 dist-test-slave-vpvm krb5kdc[29885](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332349, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.161.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:25:49.648125 30492 init.cc:377] Logged in from keytab as kudu/127.27.161.3@KRBTEST.COM (short username kudu)
I20260810 03:25:49.650242 30492 webserver.cc:460] Webserver started at http://127.27.161.3:43165/ using document root <none> and password file <none>
I20260810 03:25:49.651212 30492 fs_manager.cc:359] Metadata directory not provided
I20260810 03:25:49.651347 30492 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:25:49.651746 30492 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:25:49.655911 30492 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786332336319544-28292-0/minicluster-data/ts-2/data/instance:
uuid: "9b1c5d77b03843de81baaad1394ac7bf"
format_stamp: "Formatted at 2026-08-10 03:25:49 on dist-test-slave-vpvm"
server_key: "c4e6db0a7013c5bab3297052fba093bb"
server_key_iv: "9c063a2305c6a974719dbd3abfe94c2b"
server_key_version: "encryptionkey@0"
I20260810 03:25:49.656708 30492 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786332336319544-28292-0/minicluster-data/ts-2/wal/instance:
uuid: "9b1c5d77b03843de81baaad1394ac7bf"
format_stamp: "Formatted at 2026-08-10 03:25:49 on dist-test-slave-vpvm"
server_key: "c4e6db0a7013c5bab3297052fba093bb"
server_key_iv: "9c063a2305c6a974719dbd3abfe94c2b"
server_key_version: "encryptionkey@0"
I20260810 03:25:49.661401 30278 heartbeater.cc:503] Master 127.27.161.62:38165 was elected leader, sending a full tablet report...
I20260810 03:25:49.662478 30492 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260810 03:25:49.666568 30514 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:25:49.668038 30492 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260810 03:25:49.668207 30492 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786332336319544-28292-0/minicluster-data/ts-2/data,/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786332336319544-28292-0/minicluster-data/ts-2/wal
uuid: "9b1c5d77b03843de81baaad1394ac7bf"
format_stamp: "Formatted at 2026-08-10 03:25:49 on dist-test-slave-vpvm"
server_key: "c4e6db0a7013c5bab3297052fba093bb"
server_key_iv: "9c063a2305c6a974719dbd3abfe94c2b"
server_key_version: "encryptionkey@0"
I20260810 03:25:49.668431 30492 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786332336319544-28292-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786332336319544-28292-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786332336319544-28292-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:25:49.687494 30492 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:25:49.692037 30492 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:25:49.692323 30492 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:25:49.695348 30492 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:25:49.695467 30492 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:25:49.695626 30492 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:25:49.695708 30492 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 03:25:49 dist-test-slave-vpvm krb5kdc[29885](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332349, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.161.3@KRBTEST.COM for kudu/127.27.161.62@KRBTEST.COM
I20260810 03:25:49.721313 30322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:49.695973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.161.3:40487 (local address 127.27.161.62:38165)
0810 03:25:49.696196 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0810 03:25:49.696203 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:25:49.697810 (+ 1607us) server_negotiation.cc:407] Connection header received
0810 03:25:49.700159 (+ 2349us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:25:49.700165 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:25:49.700258 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:25:49.700346 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:25:49.702858 (+ 2512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:49.703829 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:49.706524 (+ 2695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:49.706980 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:49.707113 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:25:49.712366 (+ 5253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:25:49.712372 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:25:49.712376 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:25:49.712416 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:25:49.715597 (+ 3181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:25:49.716844 (+ 1247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:25:49.716851 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:25:49.716854 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:25:49.717039 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:25:49.717656 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:25:49.717661 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:25:49.717665 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:25:49.718084 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:25:49.718245 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0810 03:25:49.720812 (+ 2567us) server_negotiation.cc:299] Negotiation successful
0810 03:25:49.721041 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260810 03:25:49.722630 30532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:49.696671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:38165 (local address 127.27.161.3:40487)
0810 03:25:49.697618 (+ 947us) negotiation.cc:107] Waiting for socket to connect
0810 03:25:49.697662 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0810 03:25:49.699785 (+ 2123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:25:49.700824 (+ 1039us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:25:49.700840 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:25:49.701647 (+ 807us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:25:49.702635 (+ 988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:25:49.702667 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:25:49.704068 (+ 1401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:25:49.704074 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:25:49.706359 (+ 2285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:25:49.706374 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:25:49.707253 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:25:49.707263 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:25:49.707499 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:25:49.708278 (+ 779us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:25:49.708319 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:25:49.712064 (+ 3745us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:25:49.715794 (+ 3730us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:25:49.715814 (+ 20us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:25:49.715837 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:25:49.716646 (+ 809us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:25:49.717151 (+ 505us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:25:49.717157 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:25:49.717161 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:25:49.717469 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:25:49.718272 (+ 803us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:25:49.718281 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:25:49.718912 (+ 631us) client_negotiation.cc:769] Sending connection context
0810 03:25:49.719319 (+ 407us) client_negotiation.cc:240] Negotiation successful
0810 03:25:49.719969 (+ 650us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":650,"thread_start_us":390,"threads_started":1}
I20260810 03:25:49.743847 30631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:49.728846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:38165 (local address 127.27.161.3:35887)
0810 03:25:49.729689 (+ 843us) negotiation.cc:107] Waiting for socket to connect
0810 03:25:49.729713 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0810 03:25:49.729935 (+ 222us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:25:49.730335 (+ 400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:25:49.730340 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:25:49.731004 (+ 664us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:25:49.731349 (+ 345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:25:49.731359 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:25:49.732365 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:25:49.732371 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:25:49.733400 (+ 1029us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:25:49.733412 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:25:49.734377 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:25:49.734388 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:25:49.734610 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:25:49.735291 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:25:49.735317 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:25:49.736570 (+ 1253us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:25:49.740081 (+ 3511us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:25:49.740087 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:25:49.740090 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:25:49.740844 (+ 754us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:25:49.741385 (+ 541us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:25:49.741394 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:25:49.741397 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:25:49.741622 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:25:49.742264 (+ 642us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:25:49.742270 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:25:49.742538 (+ 268us) client_negotiation.cc:769] Sending connection context
0810 03:25:49.742740 (+ 202us) client_negotiation.cc:240] Negotiation successful
0810 03:25:49.743196 (+ 456us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":660,"thread_start_us":371,"threads_started":1}
I20260810 03:25:49.744922 30322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:49.728749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.161.3:35887 (local address 127.27.161.62:38165)
0810 03:25:49.728954 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0810 03:25:49.728961 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:25:49.729818 (+ 857us) server_negotiation.cc:407] Connection header received
0810 03:25:49.730135 (+ 317us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:25:49.730139 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:25:49.730207 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:25:49.730287 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:25:49.731495 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:49.732231 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:49.733535 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:49.733981 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:49.734092 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:25:49.736828 (+ 2736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:25:49.736834 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:25:49.736837 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:25:49.736875 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:25:49.739896 (+ 3021us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:25:49.741008 (+ 1112us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:25:49.741015 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:25:49.741019 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:25:49.741224 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:25:49.741772 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:25:49.741776 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:25:49.741778 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:25:49.742139 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:25:49.742299 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0810 03:25:49.744240 (+ 1941us) server_negotiation.cc:299] Negotiation successful
0810 03:25:49.744524 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260810 03:25:49.748965 30492 rpc_server.cc:307] RPC server started. Bound to: 127.27.161.3:39751
I20260810 03:25:49.748988 30692 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.161.3:39751 every 8 connection(s)
I20260810 03:25:49.750298 30492 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786332336319544-28292-0/minicluster-data/ts-2/data/info.pb
I20260810 03:25:49.751694 28292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iBH5/build/asan/bin/kudu as pid 30492
I20260810 03:25:49.751873 28292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786332336319544-28292-0/minicluster-data/ts-2/wal/instance
I20260810 03:25:49.752321 28292 external_mini_cluster.cc:1428] Setting key eeccf1205a39ef9099035a78d18ab991
I20260810 03:25:49.753808 30695 heartbeater.cc:347] Connected to a master server at 127.27.161.62:38165
I20260810 03:25:49.754076 30695 heartbeater.cc:464] Registering TS with master...
I20260810 03:25:49.754873 30695 heartbeater.cc:511] Master 127.27.161.62:38165 requested a full tablet report, sending...
I20260810 03:25:49.756240 29950 ts_manager.cc:194] Registered new tserver with Master: 9b1c5d77b03843de81baaad1394ac7bf (127.27.161.3:39751)
I20260810 03:25:49.757161 29950 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.161.3@KRBTEST.COM'} at 127.27.161.3:40487
I20260810 03:25:49.768236 28292 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 10 03:25:49 dist-test-slave-vpvm krb5kdc[29885](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332349, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 10 03:25:49 dist-test-slave-vpvm krb5kdc[29885](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332349, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.27.161.62@KRBTEST.COM
I20260810 03:25:49.825234 30322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:49.810717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56204 (local address 127.27.161.62:38165)
0810 03:25:49.810988 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0810 03:25:49.810994 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:25:49.811253 (+ 259us) server_negotiation.cc:407] Connection header received
0810 03:25:49.811446 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:25:49.811451 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:25:49.811550 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:25:49.811671 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:25:49.813161 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:49.814105 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:49.815402 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:49.815864 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:49.815987 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:25:49.819931 (+ 3944us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:25:49.819937 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:25:49.819941 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:25:49.819987 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:25:49.822648 (+ 2661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:25:49.823317 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:25:49.823322 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:25:49.823325 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:25:49.823470 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:25:49.823770 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:25:49.823773 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:25:49.823775 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:25:49.824067 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:25:49.824323 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0810 03:25:49.824697 (+ 374us) server_negotiation.cc:299] Negotiation successful
0810 03:25:49.824896 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260810 03:25:49.829512 29950 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56204:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 03:25:49.836547 29950 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 03:25:49.861080 30715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:49.849478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.1:34289 (local address 127.0.0.1:35706)
0810 03:25:49.851581 (+ 2103us) negotiation.cc:107] Waiting for socket to connect
0810 03:25:49.851602 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0810 03:25:49.851727 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:25:49.852764 (+ 1037us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:25:49.852774 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:25:49.852798 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:25:49.853272 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:25:49.853286 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:25:49.855260 (+ 1974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:25:49.855265 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:25:49.858016 (+ 2751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:25:49.858028 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:25:49.859945 (+ 1917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:25:49.859952 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:25:49.860166 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:25:49.860237 (+ 71us) client_negotiation.cc:769] Sending connection context
0810 03:25:49.860372 (+ 135us) client_negotiation.cc:240] Negotiation successful
0810 03:25:49.860585 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1965,"mutex_wait_us":180,"thread_start_us":1834,"threads_started":1}
I20260810 03:25:49.861177 30713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:49.849231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.2:44877 (local address 127.0.0.1:55652)
0810 03:25:49.850918 (+ 1687us) negotiation.cc:107] Waiting for socket to connect
0810 03:25:49.850967 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0810 03:25:49.851136 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:25:49.852398 (+ 1262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:25:49.852406 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:25:49.852438 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:25:49.852915 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:25:49.852929 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:25:49.855067 (+ 2138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:25:49.855073 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:25:49.858010 (+ 2937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:25:49.858028 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:25:49.860373 (+ 2345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:25:49.860379 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:25:49.860555 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:25:49.860586 (+ 31us) client_negotiation.cc:769] Sending connection context
0810 03:25:49.860650 (+ 64us) client_negotiation.cc:240] Negotiation successful
0810 03:25:49.860783 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1442,"mutex_wait_us":59,"thread_start_us":1290,"threads_started":1}
I20260810 03:25:49.862753 30714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:49.849483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55652 (local address 127.27.161.2:44877)
0810 03:25:49.851719 (+ 2236us) server_negotiation.cc:206] Beginning negotiation
0810 03:25:49.851729 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 03:25:49.851765 (+ 36us) server_negotiation.cc:407] Connection header received
0810 03:25:49.851881 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:25:49.851890 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:25:49.852146 (+ 256us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:25:49.853187 (+ 1041us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:25:49.853389 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:49.854910 (+ 1521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:49.858220 (+ 3310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:49.860116 (+ 1896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:49.861310 (+ 1194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:25:49.862194 (+ 884us) server_negotiation.cc:1035] Waiting for connection context
0810 03:25:49.862281 (+ 87us) server_negotiation.cc:299] Negotiation successful
0810 03:25:49.862457 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2064,"thread_start_us":1953,"threads_started":1}
I20260810 03:25:49.862756 30716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:49.850011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35706 (local address 127.27.161.1:34289)
0810 03:25:49.852190 (+ 2179us) server_negotiation.cc:206] Beginning negotiation
0810 03:25:49.852198 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 03:25:49.852226 (+ 28us) server_negotiation.cc:407] Connection header received
0810 03:25:49.852321 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:25:49.852328 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:25:49.852532 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:25:49.853470 (+ 938us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:25:49.853676 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:49.855107 (+ 1431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:49.858215 (+ 3108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:49.859698 (+ 1483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:49.861984 (+ 2286us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:25:49.862198 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0810 03:25:49.862273 (+ 75us) server_negotiation.cc:299] Negotiation successful
0810 03:25:49.862456 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2025,"thread_start_us":1913,"threads_started":1}
I20260810 03:25:49.866444 30383 tablet_service.cc:1467] Processing CreateTablet for tablet b115bcc76cb54bd193e0ac8b90b111c2 (DEFAULT_TABLE table=test-table [id=9a9a4db3290b422fa4a2bd3348bd3f87]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:25:49.866829 30168 tablet_service.cc:1467] Processing CreateTablet for tablet b115bcc76cb54bd193e0ac8b90b111c2 (DEFAULT_TABLE table=test-table [id=9a9a4db3290b422fa4a2bd3348bd3f87]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:25:49.868116 30383 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b115bcc76cb54bd193e0ac8b90b111c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:25:49.868309 30168 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b115bcc76cb54bd193e0ac8b90b111c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:25:49.871459 30715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:49.851791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.3:39751 (local address 127.0.0.1:45640)
0810 03:25:49.861637 (+ 9846us) negotiation.cc:107] Waiting for socket to connect
0810 03:25:49.861672 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0810 03:25:49.861785 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:25:49.864027 (+ 2242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:25:49.864032 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:25:49.864063 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:25:49.864242 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:25:49.864255 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:25:49.866750 (+ 2495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:25:49.866756 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:25:49.868649 (+ 1893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:25:49.868668 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:25:49.870637 (+ 1969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:25:49.870644 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:25:49.870832 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:25:49.870871 (+ 39us) client_negotiation.cc:769] Sending connection context
0810 03:25:49.870980 (+ 109us) client_negotiation.cc:240] Negotiation successful
0810 03:25:49.871141 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9739,"thread_start_us":11234,"threads_started":1}
I20260810 03:25:49.871618 30718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:49.852246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45640 (local address 127.27.161.3:39751)
0810 03:25:49.863335 (+ 11089us) server_negotiation.cc:206] Beginning negotiation
0810 03:25:49.863344 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 03:25:49.863380 (+ 36us) server_negotiation.cc:407] Connection header received
0810 03:25:49.863496 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:25:49.863516 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:25:49.863789 (+ 273us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:25:49.864454 (+ 665us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:25:49.864807 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:49.866479 (+ 1672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:49.868888 (+ 2409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:49.870421 (+ 1533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:49.870583 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:25:49.870844 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0810 03:25:49.871045 (+ 201us) server_negotiation.cc:299] Negotiation successful
0810 03:25:49.871223 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10899,"thread_start_us":10720,"threads_started":1}
I20260810 03:25:49.876225 30588 tablet_service.cc:1467] Processing CreateTablet for tablet b115bcc76cb54bd193e0ac8b90b111c2 (DEFAULT_TABLE table=test-table [id=9a9a4db3290b422fa4a2bd3348bd3f87]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:25:49.878120 30588 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b115bcc76cb54bd193e0ac8b90b111c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:25:49.882328 30720 tablet_bootstrap.cc:492] T b115bcc76cb54bd193e0ac8b90b111c2 P 0d3f86f8815e4e28a37d1c6d9385da9e: Bootstrap starting.
I20260810 03:25:49.884596 30721 tablet_bootstrap.cc:492] T b115bcc76cb54bd193e0ac8b90b111c2 P 74e398a49767421291bb348c467b01c2: Bootstrap starting.
I20260810 03:25:49.886472 30720 tablet_bootstrap.cc:654] T b115bcc76cb54bd193e0ac8b90b111c2 P 0d3f86f8815e4e28a37d1c6d9385da9e: Neither blocks nor log segments found. Creating new log.
I20260810 03:25:49.888156 30720 log.cc:824] T b115bcc76cb54bd193e0ac8b90b111c2 P 0d3f86f8815e4e28a37d1c6d9385da9e: Log is configured to *not* fsync() on all Append() calls
I20260810 03:25:49.888780 30721 tablet_bootstrap.cc:654] T b115bcc76cb54bd193e0ac8b90b111c2 P 74e398a49767421291bb348c467b01c2: Neither blocks nor log segments found. Creating new log.
I20260810 03:25:49.890504 30721 log.cc:824] T b115bcc76cb54bd193e0ac8b90b111c2 P 74e398a49767421291bb348c467b01c2: Log is configured to *not* fsync() on all Append() calls
I20260810 03:25:49.891798 30720 tablet_bootstrap.cc:492] T b115bcc76cb54bd193e0ac8b90b111c2 P 0d3f86f8815e4e28a37d1c6d9385da9e: No bootstrap required, opened a new log
I20260810 03:25:49.892190 30720 ts_tablet_manager.cc:1397] T b115bcc76cb54bd193e0ac8b90b111c2 P 0d3f86f8815e4e28a37d1c6d9385da9e: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260810 03:25:49.893855 30721 tablet_bootstrap.cc:492] T b115bcc76cb54bd193e0ac8b90b111c2 P 74e398a49767421291bb348c467b01c2: No bootstrap required, opened a new log
I20260810 03:25:49.894085 30729 tablet_bootstrap.cc:492] T b115bcc76cb54bd193e0ac8b90b111c2 P 9b1c5d77b03843de81baaad1394ac7bf: Bootstrap starting.
I20260810 03:25:49.894358 30721 ts_tablet_manager.cc:1397] T b115bcc76cb54bd193e0ac8b90b111c2 P 74e398a49767421291bb348c467b01c2: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.005s
I20260810 03:25:49.898805 30729 tablet_bootstrap.cc:654] T b115bcc76cb54bd193e0ac8b90b111c2 P 9b1c5d77b03843de81baaad1394ac7bf: Neither blocks nor log segments found. Creating new log.
I20260810 03:25:49.900473 30729 log.cc:824] T b115bcc76cb54bd193e0ac8b90b111c2 P 9b1c5d77b03843de81baaad1394ac7bf: Log is configured to *not* fsync() on all Append() calls
I20260810 03:25:49.901749 30720 raft_consensus.cc:348] T b115bcc76cb54bd193e0ac8b90b111c2 P 0d3f86f8815e4e28a37d1c6d9385da9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d3f86f8815e4e28a37d1c6d9385da9e" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 44877 } } peers { permanent_uuid: "74e398a49767421291bb348c467b01c2" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 34289 } } peers { permanent_uuid: "9b1c5d77b03843de81baaad1394ac7bf" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 39751 } }
I20260810 03:25:49.902539 30720 raft_consensus.cc:374] T b115bcc76cb54bd193e0ac8b90b111c2 P 0d3f86f8815e4e28a37d1c6d9385da9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:25:49.902678 30720 raft_consensus.cc:729] T b115bcc76cb54bd193e0ac8b90b111c2 P 0d3f86f8815e4e28a37d1c6d9385da9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d3f86f8815e4e28a37d1c6d9385da9e, State: Initialized, Role: FOLLOWER
I20260810 03:25:49.903499 30720 consensus_queue.cc:260] T b115bcc76cb54bd193e0ac8b90b111c2 P 0d3f86f8815e4e28a37d1c6d9385da9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d3f86f8815e4e28a37d1c6d9385da9e" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 44877 } } peers { permanent_uuid: "74e398a49767421291bb348c467b01c2" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 34289 } } peers { permanent_uuid: "9b1c5d77b03843de81baaad1394ac7bf" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 39751 } }
I20260810 03:25:49.904815 30486 heartbeater.cc:503] Master 127.27.161.62:38165 was elected leader, sending a full tablet report...
I20260810 03:25:49.905404 30729 tablet_bootstrap.cc:492] T b115bcc76cb54bd193e0ac8b90b111c2 P 9b1c5d77b03843de81baaad1394ac7bf: No bootstrap required, opened a new log
I20260810 03:25:49.905445 30721 raft_consensus.cc:348] T b115bcc76cb54bd193e0ac8b90b111c2 P 74e398a49767421291bb348c467b01c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d3f86f8815e4e28a37d1c6d9385da9e" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 44877 } } peers { permanent_uuid: "74e398a49767421291bb348c467b01c2" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 34289 } } peers { permanent_uuid: "9b1c5d77b03843de81baaad1394ac7bf" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 39751 } }
I20260810 03:25:49.905762 30729 ts_tablet_manager.cc:1397] T b115bcc76cb54bd193e0ac8b90b111c2 P 9b1c5d77b03843de81baaad1394ac7bf: Time spent bootstrapping tablet: real 0.012s user 0.009s sys 0.000s
I20260810 03:25:49.905905 30721 raft_consensus.cc:374] T b115bcc76cb54bd193e0ac8b90b111c2 P 74e398a49767421291bb348c467b01c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:25:49.906104 30721 raft_consensus.cc:729] T b115bcc76cb54bd193e0ac8b90b111c2 P 74e398a49767421291bb348c467b01c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74e398a49767421291bb348c467b01c2, State: Initialized, Role: FOLLOWER
I20260810 03:25:49.907037 30721 consensus_queue.cc:260] T b115bcc76cb54bd193e0ac8b90b111c2 P 74e398a49767421291bb348c467b01c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d3f86f8815e4e28a37d1c6d9385da9e" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 44877 } } peers { permanent_uuid: "74e398a49767421291bb348c467b01c2" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 34289 } } peers { permanent_uuid: "9b1c5d77b03843de81baaad1394ac7bf" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 39751 } }
I20260810 03:25:49.907621 30720 ts_tablet_manager.cc:1428] T b115bcc76cb54bd193e0ac8b90b111c2 P 0d3f86f8815e4e28a37d1c6d9385da9e: Time spent starting tablet: real 0.015s user 0.015s sys 0.000s
I20260810 03:25:49.910794 30721 ts_tablet_manager.cc:1428] T b115bcc76cb54bd193e0ac8b90b111c2 P 74e398a49767421291bb348c467b01c2: Time spent starting tablet: real 0.016s user 0.010s sys 0.005s
I20260810 03:25:49.916144 30729 raft_consensus.cc:348] T b115bcc76cb54bd193e0ac8b90b111c2 P 9b1c5d77b03843de81baaad1394ac7bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d3f86f8815e4e28a37d1c6d9385da9e" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 44877 } } peers { permanent_uuid: "74e398a49767421291bb348c467b01c2" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 34289 } } peers { permanent_uuid: "9b1c5d77b03843de81baaad1394ac7bf" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 39751 } }
I20260810 03:25:49.916589 30729 raft_consensus.cc:374] T b115bcc76cb54bd193e0ac8b90b111c2 P 9b1c5d77b03843de81baaad1394ac7bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:25:49.916668 30729 raft_consensus.cc:729] T b115bcc76cb54bd193e0ac8b90b111c2 P 9b1c5d77b03843de81baaad1394ac7bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b1c5d77b03843de81baaad1394ac7bf, State: Initialized, Role: FOLLOWER
I20260810 03:25:49.917235 30729 consensus_queue.cc:260] T b115bcc76cb54bd193e0ac8b90b111c2 P 9b1c5d77b03843de81baaad1394ac7bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d3f86f8815e4e28a37d1c6d9385da9e" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 44877 } } peers { permanent_uuid: "74e398a49767421291bb348c467b01c2" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 34289 } } peers { permanent_uuid: "9b1c5d77b03843de81baaad1394ac7bf" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 39751 } }
I20260810 03:25:49.918426 30695 heartbeater.cc:503] Master 127.27.161.62:38165 was elected leader, sending a full tablet report...
I20260810 03:25:49.919095 30729 ts_tablet_manager.cc:1428] T b115bcc76cb54bd193e0ac8b90b111c2 P 9b1c5d77b03843de81baaad1394ac7bf: Time spent starting tablet: real 0.013s user 0.005s sys 0.009s
W20260810 03:25:49.950490 30487 tablet.cc:2369] T b115bcc76cb54bd193e0ac8b90b111c2 P 0d3f86f8815e4e28a37d1c6d9385da9e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 03:25:50.002074 30696 tablet.cc:2369] T b115bcc76cb54bd193e0ac8b90b111c2 P 9b1c5d77b03843de81baaad1394ac7bf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 03:25:50.040278 30732 raft_consensus.cc:482] T b115bcc76cb54bd193e0ac8b90b111c2 P 0d3f86f8815e4e28a37d1c6d9385da9e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 03:25:50.040640 30732 raft_consensus.cc:504] T b115bcc76cb54bd193e0ac8b90b111c2 P 0d3f86f8815e4e28a37d1c6d9385da9e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d3f86f8815e4e28a37d1c6d9385da9e" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 44877 } } peers { permanent_uuid: "74e398a49767421291bb348c467b01c2" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 34289 } } peers { permanent_uuid: "9b1c5d77b03843de81baaad1394ac7bf" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 39751 } }
I20260810 03:25:50.042905 30732 leader_election.cc:288] T b115bcc76cb54bd193e0ac8b90b111c2 P 0d3f86f8815e4e28a37d1c6d9385da9e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 74e398a49767421291bb348c467b01c2 (127.27.161.1:34289), 9b1c5d77b03843de81baaad1394ac7bf (127.27.161.3:39751)
I20260810 03:25:50.050510 30716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:50.043574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.161.2:38077 (local address 127.27.161.1:34289)
0810 03:25:50.043793 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0810 03:25:50.043799 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:25:50.044293 (+ 494us) server_negotiation.cc:407] Connection header received
0810 03:25:50.044381 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:25:50.044386 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:25:50.044461 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:25:50.044560 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:25:50.045296 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:50.046352 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:50.048438 (+ 2086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:50.049609 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:50.049734 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:25:50.049796 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0810 03:25:50.050211 (+ 415us) server_negotiation.cc:299] Negotiation successful
0810 03:25:50.050315 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260810 03:25:50.050726 30737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:50.043290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.1:34289 (local address 127.27.161.2:38077)
0810 03:25:50.044065 (+ 775us) negotiation.cc:107] Waiting for socket to connect
0810 03:25:50.044085 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 03:25:50.044235 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:25:50.044749 (+ 514us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:25:50.044754 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:25:50.044787 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:25:50.045147 (+ 360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:25:50.045160 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:25:50.046485 (+ 1325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:25:50.046491 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:25:50.048258 (+ 1767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:25:50.048274 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:25:50.049837 (+ 1563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:25:50.049843 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:25:50.050043 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:25:50.050081 (+ 38us) client_negotiation.cc:769] Sending connection context
0810 03:25:50.050181 (+ 100us) client_negotiation.cc:240] Negotiation successful
0810 03:25:50.050333 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":609,"thread_start_us":405,"threads_started":1}
I20260810 03:25:50.051751 30738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:50.044246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.3:39751 (local address 127.27.161.2:60035)
0810 03:25:50.044936 (+ 690us) negotiation.cc:107] Waiting for socket to connect
0810 03:25:50.044955 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 03:25:50.045067 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:25:50.045379 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:25:50.045383 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:25:50.045407 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:25:50.045996 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:25:50.046010 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:25:50.047413 (+ 1403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:25:50.047421 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:25:50.049254 (+ 1833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:25:50.049271 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:25:50.051061 (+ 1790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:25:50.051068 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:25:50.051195 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:25:50.051221 (+ 26us) client_negotiation.cc:769] Sending connection context
0810 03:25:50.051299 (+ 78us) client_negotiation.cc:240] Negotiation successful
0810 03:25:50.051422 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":552,"thread_start_us":362,"threads_started":1}
I20260810 03:25:50.051760 30718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:50.044387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.161.2:60035 (local address 127.27.161.3:39751)
0810 03:25:50.044643 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0810 03:25:50.044649 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:25:50.045046 (+ 397us) server_negotiation.cc:407] Connection header received
0810 03:25:50.045182 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:25:50.045187 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:25:50.045271 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:25:50.045341 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:25:50.046172 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:50.047259 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:50.049436 (+ 2177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:50.050639 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:50.050810 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:25:50.050888 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0810 03:25:50.051361 (+ 473us) server_negotiation.cc:299] Negotiation successful
0810 03:25:50.051506 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260810 03:25:50.052013 30209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b115bcc76cb54bd193e0ac8b90b111c2" candidate_uuid: "0d3f86f8815e4e28a37d1c6d9385da9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74e398a49767421291bb348c467b01c2" is_pre_election: true
I20260810 03:25:50.052660 30209 raft_consensus.cc:2454] T b115bcc76cb54bd193e0ac8b90b111c2 P 74e398a49767421291bb348c467b01c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d3f86f8815e4e28a37d1c6d9385da9e in term 0.
I20260810 03:25:50.052742 30614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b115bcc76cb54bd193e0ac8b90b111c2" candidate_uuid: "0d3f86f8815e4e28a37d1c6d9385da9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b1c5d77b03843de81baaad1394ac7bf" is_pre_election: true
I20260810 03:25:50.053287 30614 raft_consensus.cc:2454] T b115bcc76cb54bd193e0ac8b90b111c2 P 9b1c5d77b03843de81baaad1394ac7bf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d3f86f8815e4e28a37d1c6d9385da9e in term 0.
I20260810 03:25:50.053399 30315 leader_election.cc:302] T b115bcc76cb54bd193e0ac8b90b111c2 P 0d3f86f8815e4e28a37d1c6d9385da9e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d3f86f8815e4e28a37d1c6d9385da9e, 74e398a49767421291bb348c467b01c2; no voters:
I20260810 03:25:50.053880 30732 raft_consensus.cc:2781] T b115bcc76cb54bd193e0ac8b90b111c2 P 0d3f86f8815e4e28a37d1c6d9385da9e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 03:25:50.054051 30732 raft_consensus.cc:482] T b115bcc76cb54bd193e0ac8b90b111c2 P 0d3f86f8815e4e28a37d1c6d9385da9e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 03:25:50.054164 30732 raft_consensus.cc:3037] T b115bcc76cb54bd193e0ac8b90b111c2 P 0d3f86f8815e4e28a37d1c6d9385da9e [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:25:50.056394 30732 raft_consensus.cc:504] T b115bcc76cb54bd193e0ac8b90b111c2 P 0d3f86f8815e4e28a37d1c6d9385da9e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d3f86f8815e4e28a37d1c6d9385da9e" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 44877 } } peers { permanent_uuid: "74e398a49767421291bb348c467b01c2" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 34289 } } peers { permanent_uuid: "9b1c5d77b03843de81baaad1394ac7bf" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 39751 } }
I20260810 03:25:50.057013 30732 leader_election.cc:288] T b115bcc76cb54bd193e0ac8b90b111c2 P 0d3f86f8815e4e28a37d1c6d9385da9e [CANDIDATE]: Term 1 election: Requested vote from peers 74e398a49767421291bb348c467b01c2 (127.27.161.1:34289), 9b1c5d77b03843de81baaad1394ac7bf (127.27.161.3:39751)
I20260810 03:25:50.057590 30209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b115bcc76cb54bd193e0ac8b90b111c2" candidate_uuid: "0d3f86f8815e4e28a37d1c6d9385da9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74e398a49767421291bb348c467b01c2"
I20260810 03:25:50.057665 30614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b115bcc76cb54bd193e0ac8b90b111c2" candidate_uuid: "0d3f86f8815e4e28a37d1c6d9385da9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b1c5d77b03843de81baaad1394ac7bf"
I20260810 03:25:50.057798 30209 raft_consensus.cc:3037] T b115bcc76cb54bd193e0ac8b90b111c2 P 74e398a49767421291bb348c467b01c2 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:25:50.057873 30614 raft_consensus.cc:3037] T b115bcc76cb54bd193e0ac8b90b111c2 P 9b1c5d77b03843de81baaad1394ac7bf [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:25:50.060477 30209 raft_consensus.cc:2454] T b115bcc76cb54bd193e0ac8b90b111c2 P 74e398a49767421291bb348c467b01c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d3f86f8815e4e28a37d1c6d9385da9e in term 1.
I20260810 03:25:50.060675 30614 raft_consensus.cc:2454] T b115bcc76cb54bd193e0ac8b90b111c2 P 9b1c5d77b03843de81baaad1394ac7bf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d3f86f8815e4e28a37d1c6d9385da9e in term 1.
I20260810 03:25:50.060993 30315 leader_election.cc:302] T b115bcc76cb54bd193e0ac8b90b111c2 P 0d3f86f8815e4e28a37d1c6d9385da9e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d3f86f8815e4e28a37d1c6d9385da9e, 74e398a49767421291bb348c467b01c2; no voters:
I20260810 03:25:50.061311 30732 raft_consensus.cc:2781] T b115bcc76cb54bd193e0ac8b90b111c2 P 0d3f86f8815e4e28a37d1c6d9385da9e [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:25:50.061873 30732 raft_consensus.cc:686] T b115bcc76cb54bd193e0ac8b90b111c2 P 0d3f86f8815e4e28a37d1c6d9385da9e [term 1 LEADER]: Becoming Leader. State: Replica: 0d3f86f8815e4e28a37d1c6d9385da9e, State: Running, Role: LEADER
I20260810 03:25:50.062601 30732 consensus_queue.cc:237] T b115bcc76cb54bd193e0ac8b90b111c2 P 0d3f86f8815e4e28a37d1c6d9385da9e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d3f86f8815e4e28a37d1c6d9385da9e" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 44877 } } peers { permanent_uuid: "74e398a49767421291bb348c467b01c2" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 34289 } } peers { permanent_uuid: "9b1c5d77b03843de81baaad1394ac7bf" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 39751 } }
I20260810 03:25:50.067786 29950 catalog_manager.cc:5495] T b115bcc76cb54bd193e0ac8b90b111c2 P 0d3f86f8815e4e28a37d1c6d9385da9e reported cstate change: term changed from 0 to 1, leader changed from <none> to 0d3f86f8815e4e28a37d1c6d9385da9e (127.27.161.2). New cstate: current_term: 1 leader_uuid: "0d3f86f8815e4e28a37d1c6d9385da9e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d3f86f8815e4e28a37d1c6d9385da9e" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 44877 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "74e398a49767421291bb348c467b01c2" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 34289 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9b1c5d77b03843de81baaad1394ac7bf" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 39751 } health_report { overall_health: UNKNOWN } } }
I20260810 03:25:50.115640 30714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:50.109320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55668 (local address 127.27.161.2:44877)
0810 03:25:50.109555 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0810 03:25:50.109562 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:25:50.109583 (+ 21us) server_negotiation.cc:407] Connection header received
0810 03:25:50.109900 (+ 317us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:25:50.109904 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:25:50.110095 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:25:50.110230 (+ 135us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 03:25:50.110638 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:50.111466 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:50.112969 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:50.113431 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:50.113547 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:25:50.114324 (+ 777us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 03:25:50.114978 (+ 654us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 03:25:50.115105 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0810 03:25:50.115261 (+ 156us) server_negotiation.cc:299] Negotiation successful
0810 03:25:50.115342 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260810 03:25:50.126226 30209 raft_consensus.cc:1261] T b115bcc76cb54bd193e0ac8b90b111c2 P 74e398a49767421291bb348c467b01c2 [term 1 FOLLOWER]: Refusing update from remote peer 0d3f86f8815e4e28a37d1c6d9385da9e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:25:50.127353 30741 consensus_queue.cc:1035] T b115bcc76cb54bd193e0ac8b90b111c2 P 0d3f86f8815e4e28a37d1c6d9385da9e [LEADER]: Connected to new peer: Peer: permanent_uuid: "74e398a49767421291bb348c467b01c2" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 34289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:25:50.128036 30614 raft_consensus.cc:1261] T b115bcc76cb54bd193e0ac8b90b111c2 P 9b1c5d77b03843de81baaad1394ac7bf [term 1 FOLLOWER]: Refusing update from remote peer 0d3f86f8815e4e28a37d1c6d9385da9e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:25:50.128882 30741 consensus_queue.cc:1035] T b115bcc76cb54bd193e0ac8b90b111c2 P 0d3f86f8815e4e28a37d1c6d9385da9e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b1c5d77b03843de81baaad1394ac7bf" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 39751 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:25:50.170357 29950 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56204:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 03:25:50.170715 29950 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56204:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 03:25:50.175618 29950 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 58a44e566ba2481daf3c673adfce18c7: Sending DeleteTablet for 3 replicas of tablet b115bcc76cb54bd193e0ac8b90b111c2
I20260810 03:25:50.177170 30588 tablet_service.cc:1514] Processing DeleteTablet for tablet b115bcc76cb54bd193e0ac8b90b111c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:25:50 UTC) from {username='kudu', principal='kudu/127.27.161.62@KRBTEST.COM'} at 127.0.0.1:45640
I20260810 03:25:50.177241 30168 tablet_service.cc:1514] Processing DeleteTablet for tablet b115bcc76cb54bd193e0ac8b90b111c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:25:50 UTC) from {username='kudu', principal='kudu/127.27.161.62@KRBTEST.COM'} at 127.0.0.1:35706
I20260810 03:25:50.177155 30383 tablet_service.cc:1514] Processing DeleteTablet for tablet b115bcc76cb54bd193e0ac8b90b111c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:25:50 UTC) from {username='kudu', principal='kudu/127.27.161.62@KRBTEST.COM'} at 127.0.0.1:55652
I20260810 03:25:50.183393 29950 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56204:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 03:25:50.185297 29950 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 03:25:50.188508 30759 tablet_replica.cc:330] T b115bcc76cb54bd193e0ac8b90b111c2 P 0d3f86f8815e4e28a37d1c6d9385da9e: stopping tablet replica
I20260810 03:25:50.189414 30759 raft_consensus.cc:2229] T b115bcc76cb54bd193e0ac8b90b111c2 P 0d3f86f8815e4e28a37d1c6d9385da9e [term 1 LEADER]: Raft consensus shutting down.
I20260810 03:25:50.195106 30759 raft_consensus.cc:2258] T b115bcc76cb54bd193e0ac8b90b111c2 P 0d3f86f8815e4e28a37d1c6d9385da9e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:25:50.196209 30756 tablet_replica.cc:330] T b115bcc76cb54bd193e0ac8b90b111c2 P 74e398a49767421291bb348c467b01c2: stopping tablet replica
I20260810 03:25:50.196741 30756 raft_consensus.cc:2229] T b115bcc76cb54bd193e0ac8b90b111c2 P 74e398a49767421291bb348c467b01c2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 03:25:50.198622 30759 ts_tablet_manager.cc:1905] T b115bcc76cb54bd193e0ac8b90b111c2 P 0d3f86f8815e4e28a37d1c6d9385da9e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:25:50.199569 30758 tablet_replica.cc:330] T b115bcc76cb54bd193e0ac8b90b111c2 P 9b1c5d77b03843de81baaad1394ac7bf: stopping tablet replica
I20260810 03:25:50.200127 30168 tablet_service.cc:1467] Processing CreateTablet for tablet 1dbf5b9330bb46999f0334a089a9ae56 (DEFAULT_TABLE table=test-table [id=c5935cd5fa7e4ebead92adc90c05a68b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:25:50.200340 30758 raft_consensus.cc:2229] T b115bcc76cb54bd193e0ac8b90b111c2 P 9b1c5d77b03843de81baaad1394ac7bf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 03:25:50.200533 30756 raft_consensus.cc:2258] T b115bcc76cb54bd193e0ac8b90b111c2 P 74e398a49767421291bb348c467b01c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:25:50.200922 30758 raft_consensus.cc:2258] T b115bcc76cb54bd193e0ac8b90b111c2 P 9b1c5d77b03843de81baaad1394ac7bf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:25:50.200914 30588 tablet_service.cc:1467] Processing CreateTablet for tablet 1dbf5b9330bb46999f0334a089a9ae56 (DEFAULT_TABLE table=test-table [id=c5935cd5fa7e4ebead92adc90c05a68b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:25:50.201287 30383 tablet_service.cc:1467] Processing CreateTablet for tablet 1dbf5b9330bb46999f0334a089a9ae56 (DEFAULT_TABLE table=test-table [id=c5935cd5fa7e4ebead92adc90c05a68b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:25:50.200860 30168 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1dbf5b9330bb46999f0334a089a9ae56. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:25:50.201807 30588 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1dbf5b9330bb46999f0334a089a9ae56. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:25:50.203040 30756 ts_tablet_manager.cc:1905] T b115bcc76cb54bd193e0ac8b90b111c2 P 74e398a49767421291bb348c467b01c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:25:50.206281 30383 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1dbf5b9330bb46999f0334a089a9ae56. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:25:50.203496 30758 ts_tablet_manager.cc:1905] T b115bcc76cb54bd193e0ac8b90b111c2 P 9b1c5d77b03843de81baaad1394ac7bf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:25:50.210491 30756 ts_tablet_manager.cc:1918] T b115bcc76cb54bd193e0ac8b90b111c2 P 74e398a49767421291bb348c467b01c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 03:25:50.210819 30721 tablet_bootstrap.cc:492] T 1dbf5b9330bb46999f0334a089a9ae56 P 74e398a49767421291bb348c467b01c2: Bootstrap starting.
I20260810 03:25:50.211087 30729 tablet_bootstrap.cc:492] T 1dbf5b9330bb46999f0334a089a9ae56 P 9b1c5d77b03843de81baaad1394ac7bf: Bootstrap starting.
I20260810 03:25:50.212723 30721 tablet_bootstrap.cc:654] T 1dbf5b9330bb46999f0334a089a9ae56 P 74e398a49767421291bb348c467b01c2: Neither blocks nor log segments found. Creating new log.
I20260810 03:25:50.212723 30729 tablet_bootstrap.cc:654] T 1dbf5b9330bb46999f0334a089a9ae56 P 9b1c5d77b03843de81baaad1394ac7bf: Neither blocks nor log segments found. Creating new log.
I20260810 03:25:50.210819 30756 log.cc:1196] T b115bcc76cb54bd193e0ac8b90b111c2 P 74e398a49767421291bb348c467b01c2: Deleting WAL directory at /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786332336319544-28292-0/minicluster-data/ts-0/wal/wals/b115bcc76cb54bd193e0ac8b90b111c2
I20260810 03:25:50.215777 30758 ts_tablet_manager.cc:1918] T b115bcc76cb54bd193e0ac8b90b111c2 P 9b1c5d77b03843de81baaad1394ac7bf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 03:25:50.216063 30758 log.cc:1196] T b115bcc76cb54bd193e0ac8b90b111c2 P 9b1c5d77b03843de81baaad1394ac7bf: Deleting WAL directory at /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786332336319544-28292-0/minicluster-data/ts-2/wal/wals/b115bcc76cb54bd193e0ac8b90b111c2
I20260810 03:25:50.216228 30756 ts_tablet_manager.cc:1939] T b115bcc76cb54bd193e0ac8b90b111c2 P 74e398a49767421291bb348c467b01c2: Deleting consensus metadata
I20260810 03:25:50.216702 30758 ts_tablet_manager.cc:1939] T b115bcc76cb54bd193e0ac8b90b111c2 P 9b1c5d77b03843de81baaad1394ac7bf: Deleting consensus metadata
I20260810 03:25:50.217590 30720 tablet_bootstrap.cc:492] T 1dbf5b9330bb46999f0334a089a9ae56 P 0d3f86f8815e4e28a37d1c6d9385da9e: Bootstrap starting.
I20260810 03:25:50.218591 29934 catalog_manager.cc:4841] TS 74e398a49767421291bb348c467b01c2 (127.27.161.1:34289): tablet b115bcc76cb54bd193e0ac8b90b111c2 (table test-table [id=9a9a4db3290b422fa4a2bd3348bd3f87]) successfully deleted
I20260810 03:25:50.219293 29934 catalog_manager.cc:4841] TS 9b1c5d77b03843de81baaad1394ac7bf (127.27.161.3:39751): tablet b115bcc76cb54bd193e0ac8b90b111c2 (table test-table [id=9a9a4db3290b422fa4a2bd3348bd3f87]) successfully deleted
I20260810 03:25:50.220933 30720 tablet_bootstrap.cc:654] T 1dbf5b9330bb46999f0334a089a9ae56 P 0d3f86f8815e4e28a37d1c6d9385da9e: Neither blocks nor log segments found. Creating new log.
I20260810 03:25:50.227921 30729 tablet_bootstrap.cc:492] T 1dbf5b9330bb46999f0334a089a9ae56 P 9b1c5d77b03843de81baaad1394ac7bf: No bootstrap required, opened a new log
I20260810 03:25:50.228088 30729 ts_tablet_manager.cc:1397] T 1dbf5b9330bb46999f0334a089a9ae56 P 9b1c5d77b03843de81baaad1394ac7bf: Time spent bootstrapping tablet: real 0.017s user 0.001s sys 0.012s
I20260810 03:25:50.228919 30729 raft_consensus.cc:348] T 1dbf5b9330bb46999f0334a089a9ae56 P 9b1c5d77b03843de81baaad1394ac7bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74e398a49767421291bb348c467b01c2" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 34289 } } peers { permanent_uuid: "0d3f86f8815e4e28a37d1c6d9385da9e" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 44877 } } peers { permanent_uuid: "9b1c5d77b03843de81baaad1394ac7bf" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 39751 } }
I20260810 03:25:50.229139 30729 raft_consensus.cc:374] T 1dbf5b9330bb46999f0334a089a9ae56 P 9b1c5d77b03843de81baaad1394ac7bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:25:50.229219 30729 raft_consensus.cc:729] T 1dbf5b9330bb46999f0334a089a9ae56 P 9b1c5d77b03843de81baaad1394ac7bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b1c5d77b03843de81baaad1394ac7bf, State: Initialized, Role: FOLLOWER
I20260810 03:25:50.229671 30729 consensus_queue.cc:260] T 1dbf5b9330bb46999f0334a089a9ae56 P 9b1c5d77b03843de81baaad1394ac7bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74e398a49767421291bb348c467b01c2" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 34289 } } peers { permanent_uuid: "0d3f86f8815e4e28a37d1c6d9385da9e" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 44877 } } peers { permanent_uuid: "9b1c5d77b03843de81baaad1394ac7bf" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 39751 } }
I20260810 03:25:50.230356 30729 ts_tablet_manager.cc:1428] T 1dbf5b9330bb46999f0334a089a9ae56 P 9b1c5d77b03843de81baaad1394ac7bf: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260810 03:25:50.231334 30721 tablet_bootstrap.cc:492] T 1dbf5b9330bb46999f0334a089a9ae56 P 74e398a49767421291bb348c467b01c2: No bootstrap required, opened a new log
I20260810 03:25:50.231580 30721 ts_tablet_manager.cc:1397] T 1dbf5b9330bb46999f0334a089a9ae56 P 74e398a49767421291bb348c467b01c2: Time spent bootstrapping tablet: real 0.021s user 0.008s sys 0.002s
I20260810 03:25:50.232439 30721 raft_consensus.cc:348] T 1dbf5b9330bb46999f0334a089a9ae56 P 74e398a49767421291bb348c467b01c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74e398a49767421291bb348c467b01c2" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 34289 } } peers { permanent_uuid: "0d3f86f8815e4e28a37d1c6d9385da9e" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 44877 } } peers { permanent_uuid: "9b1c5d77b03843de81baaad1394ac7bf" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 39751 } }
I20260810 03:25:50.232750 30721 raft_consensus.cc:374] T 1dbf5b9330bb46999f0334a089a9ae56 P 74e398a49767421291bb348c467b01c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:25:50.232889 30721 raft_consensus.cc:729] T 1dbf5b9330bb46999f0334a089a9ae56 P 74e398a49767421291bb348c467b01c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74e398a49767421291bb348c467b01c2, State: Initialized, Role: FOLLOWER
I20260810 03:25:50.233179 30721 consensus_queue.cc:260] T 1dbf5b9330bb46999f0334a089a9ae56 P 74e398a49767421291bb348c467b01c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74e398a49767421291bb348c467b01c2" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 34289 } } peers { permanent_uuid: "0d3f86f8815e4e28a37d1c6d9385da9e" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 44877 } } peers { permanent_uuid: "9b1c5d77b03843de81baaad1394ac7bf" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 39751 } }
I20260810 03:25:50.234793 30721 ts_tablet_manager.cc:1428] T 1dbf5b9330bb46999f0334a089a9ae56 P 74e398a49767421291bb348c467b01c2: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20260810 03:25:50.236372 30720 tablet_bootstrap.cc:492] T 1dbf5b9330bb46999f0334a089a9ae56 P 0d3f86f8815e4e28a37d1c6d9385da9e: No bootstrap required, opened a new log
I20260810 03:25:50.236560 30720 ts_tablet_manager.cc:1397] T 1dbf5b9330bb46999f0334a089a9ae56 P 0d3f86f8815e4e28a37d1c6d9385da9e: Time spent bootstrapping tablet: real 0.019s user 0.002s sys 0.001s
I20260810 03:25:50.237572 30720 raft_consensus.cc:348] T 1dbf5b9330bb46999f0334a089a9ae56 P 0d3f86f8815e4e28a37d1c6d9385da9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74e398a49767421291bb348c467b01c2" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 34289 } } peers { permanent_uuid: "0d3f86f8815e4e28a37d1c6d9385da9e" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 44877 } } peers { permanent_uuid: "9b1c5d77b03843de81baaad1394ac7bf" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 39751 } }
I20260810 03:25:50.237812 30720 raft_consensus.cc:374] T 1dbf5b9330bb46999f0334a089a9ae56 P 0d3f86f8815e4e28a37d1c6d9385da9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:25:50.237927 30720 raft_consensus.cc:729] T 1dbf5b9330bb46999f0334a089a9ae56 P 0d3f86f8815e4e28a37d1c6d9385da9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d3f86f8815e4e28a37d1c6d9385da9e, State: Initialized, Role: FOLLOWER
I20260810 03:25:50.238236 30720 consensus_queue.cc:260] T 1dbf5b9330bb46999f0334a089a9ae56 P 0d3f86f8815e4e28a37d1c6d9385da9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74e398a49767421291bb348c467b01c2" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 34289 } } peers { permanent_uuid: "0d3f86f8815e4e28a37d1c6d9385da9e" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 44877 } } peers { permanent_uuid: "9b1c5d77b03843de81baaad1394ac7bf" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 39751 } }
I20260810 03:25:50.238662 30759 ts_tablet_manager.cc:1918] T b115bcc76cb54bd193e0ac8b90b111c2 P 0d3f86f8815e4e28a37d1c6d9385da9e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 03:25:50.238853 30759 log.cc:1196] T b115bcc76cb54bd193e0ac8b90b111c2 P 0d3f86f8815e4e28a37d1c6d9385da9e: Deleting WAL directory at /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786332336319544-28292-0/minicluster-data/ts-1/wal/wals/b115bcc76cb54bd193e0ac8b90b111c2
I20260810 03:25:50.239543 30759 ts_tablet_manager.cc:1939] T b115bcc76cb54bd193e0ac8b90b111c2 P 0d3f86f8815e4e28a37d1c6d9385da9e: Deleting consensus metadata
I20260810 03:25:50.238857 30720 ts_tablet_manager.cc:1428] T 1dbf5b9330bb46999f0334a089a9ae56 P 0d3f86f8815e4e28a37d1c6d9385da9e: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260810 03:25:50.241470 29933 catalog_manager.cc:4841] TS 0d3f86f8815e4e28a37d1c6d9385da9e (127.27.161.2:44877): tablet b115bcc76cb54bd193e0ac8b90b111c2 (table test-table [id=9a9a4db3290b422fa4a2bd3348bd3f87]) successfully deleted
I20260810 03:25:50.347018 30733 raft_consensus.cc:482] T 1dbf5b9330bb46999f0334a089a9ae56 P 74e398a49767421291bb348c467b01c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 03:25:50.347282 30733 raft_consensus.cc:504] T 1dbf5b9330bb46999f0334a089a9ae56 P 74e398a49767421291bb348c467b01c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74e398a49767421291bb348c467b01c2" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 34289 } } peers { permanent_uuid: "0d3f86f8815e4e28a37d1c6d9385da9e" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 44877 } } peers { permanent_uuid: "9b1c5d77b03843de81baaad1394ac7bf" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 39751 } }
I20260810 03:25:50.349021 30733 leader_election.cc:288] T 1dbf5b9330bb46999f0334a089a9ae56 P 74e398a49767421291bb348c467b01c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0d3f86f8815e4e28a37d1c6d9385da9e (127.27.161.2:44877), 9b1c5d77b03843de81baaad1394ac7bf (127.27.161.3:39751)
I20260810 03:25:50.358364 30718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:50.349731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.161.1:45749 (local address 127.27.161.3:39751)
0810 03:25:50.349946 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0810 03:25:50.349978 (+ 32us) server_negotiation.cc:399] Waiting for connection header
0810 03:25:50.351053 (+ 1075us) server_negotiation.cc:407] Connection header received
0810 03:25:50.351302 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:25:50.351306 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:25:50.351371 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:25:50.351448 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:25:50.352608 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:50.353819 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:50.355883 (+ 2064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:50.357206 (+ 1323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:50.357382 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:25:50.357501 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0810 03:25:50.357915 (+ 414us) server_negotiation.cc:299] Negotiation successful
0810 03:25:50.358097 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260810 03:25:50.358644 30769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:50.349551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.3:39751 (local address 127.27.161.1:45749)
0810 03:25:50.350957 (+ 1406us) negotiation.cc:107] Waiting for socket to connect
0810 03:25:50.350972 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 03:25:50.351133 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:25:50.351614 (+ 481us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:25:50.351618 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:25:50.351641 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:25:50.352390 (+ 749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:25:50.352406 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:25:50.354061 (+ 1655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:25:50.354066 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:25:50.355684 (+ 1618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:25:50.355700 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:25:50.357504 (+ 1804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:25:50.357512 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:25:50.357709 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:25:50.357753 (+ 44us) client_negotiation.cc:769] Sending connection context
0810 03:25:50.357872 (+ 119us) client_negotiation.cc:240] Negotiation successful
0810 03:25:50.358057 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1291,"mutex_wait_us":24,"thread_start_us":590,"threads_started":1}
I20260810 03:25:50.359710 30614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1dbf5b9330bb46999f0334a089a9ae56" candidate_uuid: "74e398a49767421291bb348c467b01c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b1c5d77b03843de81baaad1394ac7bf" is_pre_election: true
I20260810 03:25:50.360041 30714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:50.350044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.161.1:41335 (local address 127.27.161.2:44877)
0810 03:25:50.350245 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0810 03:25:50.350251 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:25:50.350984 (+ 733us) server_negotiation.cc:407] Connection header received
0810 03:25:50.351226 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:25:50.351232 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:25:50.351333 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:25:50.351444 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:25:50.354067 (+ 2623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:50.355369 (+ 1302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:50.357309 (+ 1940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:50.358703 (+ 1394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:50.358884 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:25:50.358965 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0810 03:25:50.359533 (+ 568us) server_negotiation.cc:299] Negotiation successful
0810 03:25:50.359679 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260810 03:25:50.360020 30614 raft_consensus.cc:2454] T 1dbf5b9330bb46999f0334a089a9ae56 P 9b1c5d77b03843de81baaad1394ac7bf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74e398a49767421291bb348c467b01c2 in term 0.
I20260810 03:25:50.360074 30772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:50.349498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.2:44877 (local address 127.27.161.1:41335)
0810 03:25:50.350856 (+ 1358us) negotiation.cc:107] Waiting for socket to connect
0810 03:25:50.350876 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 03:25:50.351072 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:25:50.351535 (+ 463us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:25:50.351540 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:25:50.351584 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:25:50.352391 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:25:50.352406 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:25:50.355557 (+ 3151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:25:50.355564 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:25:50.357111 (+ 1547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:25:50.357127 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:25:50.359176 (+ 2049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:25:50.359184 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:25:50.359346 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:25:50.359374 (+ 28us) client_negotiation.cc:769] Sending connection context
0810 03:25:50.359492 (+ 118us) client_negotiation.cc:240] Negotiation successful
0810 03:25:50.359631 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1201,"mutex_wait_us":28,"thread_start_us":1140,"threads_started":1}
I20260810 03:25:50.360694 30093 leader_election.cc:302] T 1dbf5b9330bb46999f0334a089a9ae56 P 74e398a49767421291bb348c467b01c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74e398a49767421291bb348c467b01c2, 9b1c5d77b03843de81baaad1394ac7bf; no voters:
I20260810 03:25:50.361294 30733 raft_consensus.cc:2781] T 1dbf5b9330bb46999f0334a089a9ae56 P 74e398a49767421291bb348c467b01c2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 03:25:50.361487 30733 raft_consensus.cc:482] T 1dbf5b9330bb46999f0334a089a9ae56 P 74e398a49767421291bb348c467b01c2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 03:25:50.361617 30733 raft_consensus.cc:3037] T 1dbf5b9330bb46999f0334a089a9ae56 P 74e398a49767421291bb348c467b01c2 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:25:50.361431 30413 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1dbf5b9330bb46999f0334a089a9ae56" candidate_uuid: "74e398a49767421291bb348c467b01c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d3f86f8815e4e28a37d1c6d9385da9e" is_pre_election: true
I20260810 03:25:50.361913 30413 raft_consensus.cc:2454] T 1dbf5b9330bb46999f0334a089a9ae56 P 0d3f86f8815e4e28a37d1c6d9385da9e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74e398a49767421291bb348c467b01c2 in term 0.
I20260810 03:25:50.365160 30733 raft_consensus.cc:504] T 1dbf5b9330bb46999f0334a089a9ae56 P 74e398a49767421291bb348c467b01c2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74e398a49767421291bb348c467b01c2" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 34289 } } peers { permanent_uuid: "0d3f86f8815e4e28a37d1c6d9385da9e" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 44877 } } peers { permanent_uuid: "9b1c5d77b03843de81baaad1394ac7bf" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 39751 } }
I20260810 03:25:50.366104 30733 leader_election.cc:288] T 1dbf5b9330bb46999f0334a089a9ae56 P 74e398a49767421291bb348c467b01c2 [CANDIDATE]: Term 1 election: Requested vote from peers 0d3f86f8815e4e28a37d1c6d9385da9e (127.27.161.2:44877), 9b1c5d77b03843de81baaad1394ac7bf (127.27.161.3:39751)
I20260810 03:25:50.366760 30413 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1dbf5b9330bb46999f0334a089a9ae56" candidate_uuid: "74e398a49767421291bb348c467b01c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d3f86f8815e4e28a37d1c6d9385da9e"
I20260810 03:25:50.367017 30413 raft_consensus.cc:3037] T 1dbf5b9330bb46999f0334a089a9ae56 P 0d3f86f8815e4e28a37d1c6d9385da9e [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:25:50.366925 30614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1dbf5b9330bb46999f0334a089a9ae56" candidate_uuid: "74e398a49767421291bb348c467b01c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b1c5d77b03843de81baaad1394ac7bf"
I20260810 03:25:50.367178 30614 raft_consensus.cc:3037] T 1dbf5b9330bb46999f0334a089a9ae56 P 9b1c5d77b03843de81baaad1394ac7bf [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:25:50.369599 30413 raft_consensus.cc:2454] T 1dbf5b9330bb46999f0334a089a9ae56 P 0d3f86f8815e4e28a37d1c6d9385da9e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74e398a49767421291bb348c467b01c2 in term 1.
I20260810 03:25:50.370122 30092 leader_election.cc:302] T 1dbf5b9330bb46999f0334a089a9ae56 P 74e398a49767421291bb348c467b01c2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d3f86f8815e4e28a37d1c6d9385da9e, 74e398a49767421291bb348c467b01c2; no voters:
I20260810 03:25:50.370319 30614 raft_consensus.cc:2454] T 1dbf5b9330bb46999f0334a089a9ae56 P 9b1c5d77b03843de81baaad1394ac7bf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74e398a49767421291bb348c467b01c2 in term 1.
I20260810 03:25:50.370524 30733 raft_consensus.cc:2781] T 1dbf5b9330bb46999f0334a089a9ae56 P 74e398a49767421291bb348c467b01c2 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:25:50.371104 30733 raft_consensus.cc:686] T 1dbf5b9330bb46999f0334a089a9ae56 P 74e398a49767421291bb348c467b01c2 [term 1 LEADER]: Becoming Leader. State: Replica: 74e398a49767421291bb348c467b01c2, State: Running, Role: LEADER
I20260810 03:25:50.371857 30733 consensus_queue.cc:237] T 1dbf5b9330bb46999f0334a089a9ae56 P 74e398a49767421291bb348c467b01c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74e398a49767421291bb348c467b01c2" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 34289 } } peers { permanent_uuid: "0d3f86f8815e4e28a37d1c6d9385da9e" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 44877 } } peers { permanent_uuid: "9b1c5d77b03843de81baaad1394ac7bf" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 39751 } }
I20260810 03:25:50.375896 29949 catalog_manager.cc:5495] T 1dbf5b9330bb46999f0334a089a9ae56 P 74e398a49767421291bb348c467b01c2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 74e398a49767421291bb348c467b01c2 (127.27.161.1). New cstate: current_term: 1 leader_uuid: "74e398a49767421291bb348c467b01c2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74e398a49767421291bb348c467b01c2" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 34289 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0d3f86f8815e4e28a37d1c6d9385da9e" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 44877 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9b1c5d77b03843de81baaad1394ac7bf" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 39751 } health_report { overall_health: UNKNOWN } } }
W20260810 03:25:50.404572 30279 tablet.cc:2369] T 1dbf5b9330bb46999f0334a089a9ae56 P 74e398a49767421291bb348c467b01c2: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 10 03:25:50 dist-test-slave-vpvm krb5kdc[29885](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332347, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.161.62@KRBTEST.COM for kudu/127.27.161.62@KRBTEST.COM
I20260810 03:25:50.448813 30786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:50.432611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56212 (local address 127.27.161.62:38165)
0810 03:25:50.433424 (+ 813us) server_negotiation.cc:206] Beginning negotiation
0810 03:25:50.433434 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 03:25:50.433689 (+ 255us) server_negotiation.cc:407] Connection header received
0810 03:25:50.433866 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:25:50.433870 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:25:50.433933 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:25:50.434056 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:25:50.435199 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:50.436120 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:50.437611 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:50.438191 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:50.438324 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:25:50.442161 (+ 3837us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:25:50.442167 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:25:50.442170 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:25:50.442204 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:25:50.445201 (+ 2997us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:25:50.446464 (+ 1263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:25:50.446469 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:25:50.446471 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:25:50.446628 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:25:50.447090 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:25:50.447094 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:25:50.447096 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:25:50.447393 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:25:50.447526 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0810 03:25:50.448097 (+ 571us) server_negotiation.cc:299] Negotiation successful
0810 03:25:50.448450 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":150,"server-negotiator.queue_time_us":633,"thread_start_us":452,"threads_started":1}
I20260810 03:25:50.448834 30788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:50.432710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:38165 (local address 127.0.0.1:56212)
0810 03:25:50.433587 (+ 877us) negotiation.cc:107] Waiting for socket to connect
0810 03:25:50.433602 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 03:25:50.433754 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:25:50.434080 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:25:50.434084 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:25:50.434804 (+ 720us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:25:50.435036 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:25:50.435047 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:25:50.436258 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:25:50.436263 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:25:50.437447 (+ 1184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:25:50.437466 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:25:50.438391 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:25:50.438397 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:25:50.438592 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:25:50.439246 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:25:50.439265 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:25:50.441886 (+ 2621us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:25:50.445377 (+ 3491us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:25:50.445396 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:25:50.445405 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:25:50.446314 (+ 909us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:25:50.446758 (+ 444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:25:50.446763 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:25:50.446767 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:25:50.446955 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:25:50.447537 (+ 582us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:25:50.447547 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:25:50.447770 (+ 223us) client_negotiation.cc:769] Sending connection context
0810 03:25:50.447977 (+ 207us) client_negotiation.cc:240] Negotiation successful
0810 03:25:50.448324 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":760,"thread_start_us":599,"threads_started":1}
I20260810 03:25:50.455449 29949 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.27.161.62@KRBTEST.COM'} at 127.0.0.1:56212:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260810 03:25:50.457154 29949 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 03:25:50.465199 30168 tablet_service.cc:1467] Processing CreateTablet for tablet 5cf479ea96d24ffd9516743976df63b4 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=7b80caac477c41728afbff05431ace5a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 03:25:50.465225 30383 tablet_service.cc:1467] Processing CreateTablet for tablet 5cf479ea96d24ffd9516743976df63b4 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=7b80caac477c41728afbff05431ace5a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 03:25:50.465842 30383 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5cf479ea96d24ffd9516743976df63b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:25:50.465835 30168 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5cf479ea96d24ffd9516743976df63b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:25:50.465706 30588 tablet_service.cc:1467] Processing CreateTablet for tablet 5cf479ea96d24ffd9516743976df63b4 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=7b80caac477c41728afbff05431ace5a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 03:25:50.466344 30588 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5cf479ea96d24ffd9516743976df63b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:25:50.474112 30720 tablet_bootstrap.cc:492] T 5cf479ea96d24ffd9516743976df63b4 P 0d3f86f8815e4e28a37d1c6d9385da9e: Bootstrap starting.
I20260810 03:25:50.475111 30729 tablet_bootstrap.cc:492] T 5cf479ea96d24ffd9516743976df63b4 P 9b1c5d77b03843de81baaad1394ac7bf: Bootstrap starting.
I20260810 03:25:50.475507 30720 tablet_bootstrap.cc:654] T 5cf479ea96d24ffd9516743976df63b4 P 0d3f86f8815e4e28a37d1c6d9385da9e: Neither blocks nor log segments found. Creating new log.
I20260810 03:25:50.476495 30721 tablet_bootstrap.cc:492] T 5cf479ea96d24ffd9516743976df63b4 P 74e398a49767421291bb348c467b01c2: Bootstrap starting.
I20260810 03:25:50.477057 30729 tablet_bootstrap.cc:654] T 5cf479ea96d24ffd9516743976df63b4 P 9b1c5d77b03843de81baaad1394ac7bf: Neither blocks nor log segments found. Creating new log.
I20260810 03:25:50.478736 30721 tablet_bootstrap.cc:654] T 5cf479ea96d24ffd9516743976df63b4 P 74e398a49767421291bb348c467b01c2: Neither blocks nor log segments found. Creating new log.
I20260810 03:25:50.480849 30729 tablet_bootstrap.cc:492] T 5cf479ea96d24ffd9516743976df63b4 P 9b1c5d77b03843de81baaad1394ac7bf: No bootstrap required, opened a new log
I20260810 03:25:50.481063 30729 ts_tablet_manager.cc:1397] T 5cf479ea96d24ffd9516743976df63b4 P 9b1c5d77b03843de81baaad1394ac7bf: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20260810 03:25:50.483461 30729 raft_consensus.cc:348] T 5cf479ea96d24ffd9516743976df63b4 P 9b1c5d77b03843de81baaad1394ac7bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74e398a49767421291bb348c467b01c2" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 34289 } } peers { permanent_uuid: "0d3f86f8815e4e28a37d1c6d9385da9e" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 44877 } } peers { permanent_uuid: "9b1c5d77b03843de81baaad1394ac7bf" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 39751 } }
I20260810 03:25:50.483742 30729 raft_consensus.cc:374] T 5cf479ea96d24ffd9516743976df63b4 P 9b1c5d77b03843de81baaad1394ac7bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:25:50.483832 30729 raft_consensus.cc:729] T 5cf479ea96d24ffd9516743976df63b4 P 9b1c5d77b03843de81baaad1394ac7bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b1c5d77b03843de81baaad1394ac7bf, State: Initialized, Role: FOLLOWER
I20260810 03:25:50.484146 30729 consensus_queue.cc:260] T 5cf479ea96d24ffd9516743976df63b4 P 9b1c5d77b03843de81baaad1394ac7bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74e398a49767421291bb348c467b01c2" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 34289 } } peers { permanent_uuid: "0d3f86f8815e4e28a37d1c6d9385da9e" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 44877 } } peers { permanent_uuid: "9b1c5d77b03843de81baaad1394ac7bf" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 39751 } }
I20260810 03:25:50.484781 30736 tablet_replica.cc:439] T 5cf479ea96d24ffd9516743976df63b4 P 9b1c5d77b03843de81baaad1394ac7bf: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74e398a49767421291bb348c467b01c2" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 34289 } } peers { permanent_uuid: "0d3f86f8815e4e28a37d1c6d9385da9e" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 44877 } } peers { permanent_uuid: "9b1c5d77b03843de81baaad1394ac7bf" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 39751 } } }
I20260810 03:25:50.485152 30736 tablet_replica.cc:442] T 5cf479ea96d24ffd9516743976df63b4 P 9b1c5d77b03843de81baaad1394ac7bf: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 03:25:50.484783 30729 ts_tablet_manager.cc:1428] T 5cf479ea96d24ffd9516743976df63b4 P 9b1c5d77b03843de81baaad1394ac7bf: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260810 03:25:50.487972 30720 tablet_bootstrap.cc:492] T 5cf479ea96d24ffd9516743976df63b4 P 0d3f86f8815e4e28a37d1c6d9385da9e: No bootstrap required, opened a new log
I20260810 03:25:50.488240 30720 ts_tablet_manager.cc:1397] T 5cf479ea96d24ffd9516743976df63b4 P 0d3f86f8815e4e28a37d1c6d9385da9e: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.006s
I20260810 03:25:50.489233 30720 raft_consensus.cc:348] T 5cf479ea96d24ffd9516743976df63b4 P 0d3f86f8815e4e28a37d1c6d9385da9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74e398a49767421291bb348c467b01c2" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 34289 } } peers { permanent_uuid: "0d3f86f8815e4e28a37d1c6d9385da9e" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 44877 } } peers { permanent_uuid: "9b1c5d77b03843de81baaad1394ac7bf" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 39751 } }
I20260810 03:25:50.489564 30720 raft_consensus.cc:374] T 5cf479ea96d24ffd9516743976df63b4 P 0d3f86f8815e4e28a37d1c6d9385da9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:25:50.489693 30720 raft_consensus.cc:729] T 5cf479ea96d24ffd9516743976df63b4 P 0d3f86f8815e4e28a37d1c6d9385da9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d3f86f8815e4e28a37d1c6d9385da9e, State: Initialized, Role: FOLLOWER
I20260810 03:25:50.490101 30720 consensus_queue.cc:260] T 5cf479ea96d24ffd9516743976df63b4 P 0d3f86f8815e4e28a37d1c6d9385da9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74e398a49767421291bb348c467b01c2" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 34289 } } peers { permanent_uuid: "0d3f86f8815e4e28a37d1c6d9385da9e" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 44877 } } peers { permanent_uuid: "9b1c5d77b03843de81baaad1394ac7bf" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 39751 } }
I20260810 03:25:50.490595 30732 tablet_replica.cc:439] T 5cf479ea96d24ffd9516743976df63b4 P 0d3f86f8815e4e28a37d1c6d9385da9e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74e398a49767421291bb348c467b01c2" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 34289 } } peers { permanent_uuid: "0d3f86f8815e4e28a37d1c6d9385da9e" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 44877 } } peers { permanent_uuid: "9b1c5d77b03843de81baaad1394ac7bf" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 39751 } } }
I20260810 03:25:50.490936 30732 tablet_replica.cc:442] T 5cf479ea96d24ffd9516743976df63b4 P 0d3f86f8815e4e28a37d1c6d9385da9e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 03:25:50.491577 30721 tablet_bootstrap.cc:492] T 5cf479ea96d24ffd9516743976df63b4 P 74e398a49767421291bb348c467b01c2: No bootstrap required, opened a new log
I20260810 03:25:50.491760 30721 ts_tablet_manager.cc:1397] T 5cf479ea96d24ffd9516743976df63b4 P 74e398a49767421291bb348c467b01c2: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.005s
I20260810 03:25:50.492374 30720 ts_tablet_manager.cc:1428] T 5cf479ea96d24ffd9516743976df63b4 P 0d3f86f8815e4e28a37d1c6d9385da9e: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260810 03:25:50.492671 30721 raft_consensus.cc:348] T 5cf479ea96d24ffd9516743976df63b4 P 74e398a49767421291bb348c467b01c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74e398a49767421291bb348c467b01c2" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 34289 } } peers { permanent_uuid: "0d3f86f8815e4e28a37d1c6d9385da9e" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 44877 } } peers { permanent_uuid: "9b1c5d77b03843de81baaad1394ac7bf" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 39751 } }
I20260810 03:25:50.492885 30721 raft_consensus.cc:374] T 5cf479ea96d24ffd9516743976df63b4 P 74e398a49767421291bb348c467b01c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:25:50.492974 30721 raft_consensus.cc:729] T 5cf479ea96d24ffd9516743976df63b4 P 74e398a49767421291bb348c467b01c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74e398a49767421291bb348c467b01c2, State: Initialized, Role: FOLLOWER
I20260810 03:25:50.493268 30721 consensus_queue.cc:260] T 5cf479ea96d24ffd9516743976df63b4 P 74e398a49767421291bb348c467b01c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74e398a49767421291bb348c467b01c2" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 34289 } } peers { permanent_uuid: "0d3f86f8815e4e28a37d1c6d9385da9e" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 44877 } } peers { permanent_uuid: "9b1c5d77b03843de81baaad1394ac7bf" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 39751 } }
I20260810 03:25:50.493989 30721 ts_tablet_manager.cc:1428] T 5cf479ea96d24ffd9516743976df63b4 P 74e398a49767421291bb348c467b01c2: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260810 03:25:50.493796 30733 tablet_replica.cc:439] T 5cf479ea96d24ffd9516743976df63b4 P 74e398a49767421291bb348c467b01c2: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74e398a49767421291bb348c467b01c2" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 34289 } } peers { permanent_uuid: "0d3f86f8815e4e28a37d1c6d9385da9e" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 44877 } } peers { permanent_uuid: "9b1c5d77b03843de81baaad1394ac7bf" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 39751 } } }
I20260810 03:25:50.494167 30733 tablet_replica.cc:442] T 5cf479ea96d24ffd9516743976df63b4 P 74e398a49767421291bb348c467b01c2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 03:25:50.720055 30733 raft_consensus.cc:482] T 5cf479ea96d24ffd9516743976df63b4 P 74e398a49767421291bb348c467b01c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 03:25:50.720265 30733 raft_consensus.cc:504] T 5cf479ea96d24ffd9516743976df63b4 P 74e398a49767421291bb348c467b01c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74e398a49767421291bb348c467b01c2" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 34289 } } peers { permanent_uuid: "0d3f86f8815e4e28a37d1c6d9385da9e" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 44877 } } peers { permanent_uuid: "9b1c5d77b03843de81baaad1394ac7bf" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 39751 } }
I20260810 03:25:50.720858 30733 leader_election.cc:288] T 5cf479ea96d24ffd9516743976df63b4 P 74e398a49767421291bb348c467b01c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0d3f86f8815e4e28a37d1c6d9385da9e (127.27.161.2:44877), 9b1c5d77b03843de81baaad1394ac7bf (127.27.161.3:39751)
I20260810 03:25:50.721455 30614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5cf479ea96d24ffd9516743976df63b4" candidate_uuid: "74e398a49767421291bb348c467b01c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b1c5d77b03843de81baaad1394ac7bf" is_pre_election: true
I20260810 03:25:50.721457 30413 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5cf479ea96d24ffd9516743976df63b4" candidate_uuid: "74e398a49767421291bb348c467b01c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d3f86f8815e4e28a37d1c6d9385da9e" is_pre_election: true
I20260810 03:25:50.721714 30614 raft_consensus.cc:2454] T 5cf479ea96d24ffd9516743976df63b4 P 9b1c5d77b03843de81baaad1394ac7bf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74e398a49767421291bb348c467b01c2 in term 0.
I20260810 03:25:50.721736 30413 raft_consensus.cc:2454] T 5cf479ea96d24ffd9516743976df63b4 P 0d3f86f8815e4e28a37d1c6d9385da9e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74e398a49767421291bb348c467b01c2 in term 0.
I20260810 03:25:50.722157 30093 leader_election.cc:302] T 5cf479ea96d24ffd9516743976df63b4 P 74e398a49767421291bb348c467b01c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74e398a49767421291bb348c467b01c2, 9b1c5d77b03843de81baaad1394ac7bf; no voters:
I20260810 03:25:50.722450 30733 raft_consensus.cc:2781] T 5cf479ea96d24ffd9516743976df63b4 P 74e398a49767421291bb348c467b01c2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 03:25:50.722543 30733 raft_consensus.cc:482] T 5cf479ea96d24ffd9516743976df63b4 P 74e398a49767421291bb348c467b01c2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 03:25:50.722659 30733 raft_consensus.cc:3037] T 5cf479ea96d24ffd9516743976df63b4 P 74e398a49767421291bb348c467b01c2 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:25:50.724838 30733 raft_consensus.cc:504] T 5cf479ea96d24ffd9516743976df63b4 P 74e398a49767421291bb348c467b01c2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74e398a49767421291bb348c467b01c2" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 34289 } } peers { permanent_uuid: "0d3f86f8815e4e28a37d1c6d9385da9e" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 44877 } } peers { permanent_uuid: "9b1c5d77b03843de81baaad1394ac7bf" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 39751 } }
I20260810 03:25:50.725507 30733 leader_election.cc:288] T 5cf479ea96d24ffd9516743976df63b4 P 74e398a49767421291bb348c467b01c2 [CANDIDATE]: Term 1 election: Requested vote from peers 0d3f86f8815e4e28a37d1c6d9385da9e (127.27.161.2:44877), 9b1c5d77b03843de81baaad1394ac7bf (127.27.161.3:39751)
I20260810 03:25:50.725999 30413 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5cf479ea96d24ffd9516743976df63b4" candidate_uuid: "74e398a49767421291bb348c467b01c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d3f86f8815e4e28a37d1c6d9385da9e"
I20260810 03:25:50.726110 30614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5cf479ea96d24ffd9516743976df63b4" candidate_uuid: "74e398a49767421291bb348c467b01c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b1c5d77b03843de81baaad1394ac7bf"
I20260810 03:25:50.726280 30413 raft_consensus.cc:3037] T 5cf479ea96d24ffd9516743976df63b4 P 0d3f86f8815e4e28a37d1c6d9385da9e [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:25:50.726342 30614 raft_consensus.cc:3037] T 5cf479ea96d24ffd9516743976df63b4 P 9b1c5d77b03843de81baaad1394ac7bf [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:25:50.729188 30413 raft_consensus.cc:2454] T 5cf479ea96d24ffd9516743976df63b4 P 0d3f86f8815e4e28a37d1c6d9385da9e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74e398a49767421291bb348c467b01c2 in term 1.
I20260810 03:25:50.729189 30614 raft_consensus.cc:2454] T 5cf479ea96d24ffd9516743976df63b4 P 9b1c5d77b03843de81baaad1394ac7bf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74e398a49767421291bb348c467b01c2 in term 1.
I20260810 03:25:50.729682 30092 leader_election.cc:302] T 5cf479ea96d24ffd9516743976df63b4 P 74e398a49767421291bb348c467b01c2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d3f86f8815e4e28a37d1c6d9385da9e, 74e398a49767421291bb348c467b01c2; no voters:
I20260810 03:25:50.730016 30733 raft_consensus.cc:2781] T 5cf479ea96d24ffd9516743976df63b4 P 74e398a49767421291bb348c467b01c2 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:25:50.730208 30733 raft_consensus.cc:686] T 5cf479ea96d24ffd9516743976df63b4 P 74e398a49767421291bb348c467b01c2 [term 1 LEADER]: Becoming Leader. State: Replica: 74e398a49767421291bb348c467b01c2, State: Running, Role: LEADER
I20260810 03:25:50.730463 30733 consensus_queue.cc:237] T 5cf479ea96d24ffd9516743976df63b4 P 74e398a49767421291bb348c467b01c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74e398a49767421291bb348c467b01c2" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 34289 } } peers { permanent_uuid: "0d3f86f8815e4e28a37d1c6d9385da9e" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 44877 } } peers { permanent_uuid: "9b1c5d77b03843de81baaad1394ac7bf" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 39751 } }
I20260810 03:25:50.732378 30775 tablet_replica.cc:439] T 5cf479ea96d24ffd9516743976df63b4 P 74e398a49767421291bb348c467b01c2: TxnStatusTablet state changed. Reason: New leader 74e398a49767421291bb348c467b01c2. Latest consensus state: current_term: 1 leader_uuid: "74e398a49767421291bb348c467b01c2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74e398a49767421291bb348c467b01c2" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 34289 } } peers { permanent_uuid: "0d3f86f8815e4e28a37d1c6d9385da9e" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 44877 } } peers { permanent_uuid: "9b1c5d77b03843de81baaad1394ac7bf" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 39751 } } }
I20260810 03:25:50.732674 30775 tablet_replica.cc:442] T 5cf479ea96d24ffd9516743976df63b4 P 74e398a49767421291bb348c467b01c2: This TxnStatusTablet replica's current role is: LEADER
I20260810 03:25:50.734031 30793 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260810 03:25:50.733775 29951 catalog_manager.cc:5495] T 5cf479ea96d24ffd9516743976df63b4 P 74e398a49767421291bb348c467b01c2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 74e398a49767421291bb348c467b01c2 (127.27.161.1). New cstate: current_term: 1 leader_uuid: "74e398a49767421291bb348c467b01c2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74e398a49767421291bb348c467b01c2" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 34289 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0d3f86f8815e4e28a37d1c6d9385da9e" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 44877 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9b1c5d77b03843de81baaad1394ac7bf" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 39751 } health_report { overall_health: UNKNOWN } } }
I20260810 03:25:50.734351 30793 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260810 03:25:50.790153 30788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:50.783872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.1:34289 (local address 127.0.0.1:35714)
0810 03:25:50.784130 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0810 03:25:50.784154 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0810 03:25:50.784311 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:25:50.784969 (+ 658us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:25:50.784975 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:25:50.784988 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 03:25:50.785212 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:25:50.785223 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:25:50.786482 (+ 1259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:25:50.786486 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:25:50.787519 (+ 1033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:25:50.787537 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:25:50.788316 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:25:50.788323 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:25:50.788475 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:25:50.788552 (+ 77us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 03:25:50.789502 (+ 950us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 03:25:50.789531 (+ 29us) client_negotiation.cc:769] Sending connection context
0810 03:25:50.789619 (+ 88us) client_negotiation.cc:240] Negotiation successful
0810 03:25:50.789786 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260810 03:25:50.790215 30796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:50.783933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35714 (local address 127.27.161.1:34289)
0810 03:25:50.784597 (+ 664us) server_negotiation.cc:206] Beginning negotiation
0810 03:25:50.784602 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:25:50.784617 (+ 15us) server_negotiation.cc:407] Connection header received
0810 03:25:50.784703 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:25:50.784706 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:25:50.784780 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:25:50.784921 (+ 141us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 03:25:50.785384 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:50.786335 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:50.787681 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:50.788155 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:50.788271 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:25:50.788768 (+ 497us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 03:25:50.789386 (+ 618us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 03:25:50.789547 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0810 03:25:50.789674 (+ 127us) server_negotiation.cc:299] Negotiation successful
0810 03:25:50.789772 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":540,"thread_start_us":316,"threads_started":1}
I20260810 03:25:50.796015 30614 raft_consensus.cc:1261] T 5cf479ea96d24ffd9516743976df63b4 P 9b1c5d77b03843de81baaad1394ac7bf [term 1 FOLLOWER]: Refusing update from remote peer 74e398a49767421291bb348c467b01c2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:25:50.796340 30413 raft_consensus.cc:1261] T 5cf479ea96d24ffd9516743976df63b4 P 0d3f86f8815e4e28a37d1c6d9385da9e [term 1 FOLLOWER]: Refusing update from remote peer 74e398a49767421291bb348c467b01c2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:25:50.796787 30775 consensus_queue.cc:1035] T 5cf479ea96d24ffd9516743976df63b4 P 74e398a49767421291bb348c467b01c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b1c5d77b03843de81baaad1394ac7bf" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 39751 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:25:50.797178 30733 consensus_queue.cc:1035] T 5cf479ea96d24ffd9516743976df63b4 P 74e398a49767421291bb348c467b01c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d3f86f8815e4e28a37d1c6d9385da9e" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 44877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:25:50.804077 30736 tablet_replica.cc:439] T 5cf479ea96d24ffd9516743976df63b4 P 9b1c5d77b03843de81baaad1394ac7bf: TxnStatusTablet state changed. Reason: New leader 74e398a49767421291bb348c467b01c2. Latest consensus state: current_term: 1 leader_uuid: "74e398a49767421291bb348c467b01c2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74e398a49767421291bb348c467b01c2" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 34289 } } peers { permanent_uuid: "0d3f86f8815e4e28a37d1c6d9385da9e" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 44877 } } peers { permanent_uuid: "9b1c5d77b03843de81baaad1394ac7bf" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 39751 } } }
I20260810 03:25:50.804422 30736 tablet_replica.cc:442] T 5cf479ea96d24ffd9516743976df63b4 P 9b1c5d77b03843de81baaad1394ac7bf: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 03:25:50.806819 30797 mvcc.cc:205] Tried to move back new op lower bound from 7316817308852678656 to 7316817308594368512. Current Snapshot: MvccSnapshot[applied={T|T < 7316817308852678656}]
I20260810 03:25:50.808097 30775 tablet_replica.cc:439] T 5cf479ea96d24ffd9516743976df63b4 P 74e398a49767421291bb348c467b01c2: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "74e398a49767421291bb348c467b01c2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74e398a49767421291bb348c467b01c2" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 34289 } } peers { permanent_uuid: "0d3f86f8815e4e28a37d1c6d9385da9e" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 44877 } } peers { permanent_uuid: "9b1c5d77b03843de81baaad1394ac7bf" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 39751 } } }
I20260810 03:25:50.808398 30775 tablet_replica.cc:442] T 5cf479ea96d24ffd9516743976df63b4 P 74e398a49767421291bb348c467b01c2: This TxnStatusTablet replica's current role is: LEADER
I20260810 03:25:50.812269 30733 consensus_queue.cc:1035] T 1dbf5b9330bb46999f0334a089a9ae56 P 74e398a49767421291bb348c467b01c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b1c5d77b03843de81baaad1394ac7bf" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 39751 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:25:50.820459 30775 consensus_queue.cc:1035] T 1dbf5b9330bb46999f0334a089a9ae56 P 74e398a49767421291bb348c467b01c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d3f86f8815e4e28a37d1c6d9385da9e" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 44877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:25:50.842555 30736 tablet_replica.cc:439] T 5cf479ea96d24ffd9516743976df63b4 P 9b1c5d77b03843de81baaad1394ac7bf: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "74e398a49767421291bb348c467b01c2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74e398a49767421291bb348c467b01c2" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 34289 } } peers { permanent_uuid: "0d3f86f8815e4e28a37d1c6d9385da9e" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 44877 } } peers { permanent_uuid: "9b1c5d77b03843de81baaad1394ac7bf" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 39751 } } }
I20260810 03:25:50.842871 30736 tablet_replica.cc:442] T 5cf479ea96d24ffd9516743976df63b4 P 9b1c5d77b03843de81baaad1394ac7bf: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 03:25:50.842800 30732 tablet_replica.cc:439] T 5cf479ea96d24ffd9516743976df63b4 P 0d3f86f8815e4e28a37d1c6d9385da9e: TxnStatusTablet state changed. Reason: New leader 74e398a49767421291bb348c467b01c2. Latest consensus state: current_term: 1 leader_uuid: "74e398a49767421291bb348c467b01c2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74e398a49767421291bb348c467b01c2" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 34289 } } peers { permanent_uuid: "0d3f86f8815e4e28a37d1c6d9385da9e" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 44877 } } peers { permanent_uuid: "9b1c5d77b03843de81baaad1394ac7bf" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 39751 } } }
I20260810 03:25:50.843120 30732 tablet_replica.cc:442] T 5cf479ea96d24ffd9516743976df63b4 P 0d3f86f8815e4e28a37d1c6d9385da9e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 03:25:50.844611 30775 tablet_replica.cc:439] T 5cf479ea96d24ffd9516743976df63b4 P 74e398a49767421291bb348c467b01c2: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "74e398a49767421291bb348c467b01c2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74e398a49767421291bb348c467b01c2" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 34289 } } peers { permanent_uuid: "0d3f86f8815e4e28a37d1c6d9385da9e" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 44877 } } peers { permanent_uuid: "9b1c5d77b03843de81baaad1394ac7bf" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 39751 } } }
I20260810 03:25:50.844935 30775 tablet_replica.cc:442] T 5cf479ea96d24ffd9516743976df63b4 P 74e398a49767421291bb348c467b01c2: This TxnStatusTablet replica's current role is: LEADER
I20260810 03:25:50.851402 30732 tablet_replica.cc:439] T 5cf479ea96d24ffd9516743976df63b4 P 0d3f86f8815e4e28a37d1c6d9385da9e: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "74e398a49767421291bb348c467b01c2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74e398a49767421291bb348c467b01c2" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 34289 } } peers { permanent_uuid: "0d3f86f8815e4e28a37d1c6d9385da9e" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 44877 } } peers { permanent_uuid: "9b1c5d77b03843de81baaad1394ac7bf" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 39751 } } }
I20260810 03:25:50.851771 30732 tablet_replica.cc:442] T 5cf479ea96d24ffd9516743976df63b4 P 0d3f86f8815e4e28a37d1c6d9385da9e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 03:25:50.852648 30796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:50.823633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35730 (local address 127.27.161.1:34289)
0810 03:25:50.823813 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0810 03:25:50.823819 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:25:50.836617 (+ 12798us) server_negotiation.cc:407] Connection header received
0810 03:25:50.837731 (+ 1114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:25:50.837738 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:25:50.837837 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:25:50.837902 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 03:25:50.846130 (+ 8228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:50.847328 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:50.849499 (+ 2171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:50.850362 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:50.850470 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:25:50.851181 (+ 711us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 03:25:50.851324 (+ 143us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 03:25:50.851401 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0810 03:25:50.851975 (+ 574us) server_negotiation.cc:299] Negotiation successful
0810 03:25:50.852157 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260810 03:25:50.874044 30818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:50.867273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.1:34289 (local address 127.27.161.1:39975)
0810 03:25:50.868338 (+ 1065us) negotiation.cc:107] Waiting for socket to connect
0810 03:25:50.868355 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 03:25:50.868520 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:25:50.868856 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:25:50.868860 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:25:50.868871 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 03:25:50.869240 (+ 369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:25:50.869253 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:25:50.870624 (+ 1371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:25:50.870634 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:25:50.871793 (+ 1159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:25:50.871807 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:25:50.872673 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:25:50.872679 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:25:50.872815 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:25:50.872878 (+ 63us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 03:25:50.873267 (+ 389us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 03:25:50.873281 (+ 14us) client_negotiation.cc:769] Sending connection context
0810 03:25:50.873353 (+ 72us) client_negotiation.cc:240] Negotiation successful
0810 03:25:50.873559 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":882,"mutex_wait_us":1,"thread_start_us":644,"threads_started":1}
I20260810 03:25:50.874039 30796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:50.867281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.161.1:39975 (local address 127.27.161.1:34289)
0810 03:25:50.867474 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0810 03:25:50.867481 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:25:50.868443 (+ 962us) server_negotiation.cc:407] Connection header received
0810 03:25:50.868669 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:25:50.868675 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:25:50.868752 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:25:50.868826 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 03:25:50.869393 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:50.870454 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:50.871939 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:50.872512 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:50.872631 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:25:50.873053 (+ 422us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 03:25:50.873175 (+ 122us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 03:25:50.873249 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0810 03:25:50.873387 (+ 138us) server_negotiation.cc:299] Negotiation successful
0810 03:25:50.873498 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":25,"server-negotiator.queue_time_us":58}
I20260810 03:25:50.954702 29951 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56204:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 03:25:50.954980 29951 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56204:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 03:25:50.958806 29951 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 58a44e566ba2481daf3c673adfce18c7: Sending DeleteTablet for 3 replicas of tablet 1dbf5b9330bb46999f0334a089a9ae56
I20260810 03:25:50.959941 30383 tablet_service.cc:1514] Processing DeleteTablet for tablet 1dbf5b9330bb46999f0334a089a9ae56 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:25:50 UTC) from {username='kudu', principal='kudu/127.27.161.62@KRBTEST.COM'} at 127.0.0.1:55652
I20260810 03:25:50.960227 30588 tablet_service.cc:1514] Processing DeleteTablet for tablet 1dbf5b9330bb46999f0334a089a9ae56 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:25:50 UTC) from {username='kudu', principal='kudu/127.27.161.62@KRBTEST.COM'} at 127.0.0.1:45640
I20260810 03:25:50.961140 30168 tablet_service.cc:1514] Processing DeleteTablet for tablet 1dbf5b9330bb46999f0334a089a9ae56 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:25:50 UTC) from {username='kudu', principal='kudu/127.27.161.62@KRBTEST.COM'} at 127.0.0.1:35706
I20260810 03:25:50.967140 30826 tablet_replica.cc:330] T 1dbf5b9330bb46999f0334a089a9ae56 P 74e398a49767421291bb348c467b01c2: stopping tablet replica
I20260810 03:25:50.968098 30826 raft_consensus.cc:2229] T 1dbf5b9330bb46999f0334a089a9ae56 P 74e398a49767421291bb348c467b01c2 [term 1 LEADER]: Raft consensus shutting down.
I20260810 03:25:50.969132 30826 raft_consensus.cc:2258] T 1dbf5b9330bb46999f0334a089a9ae56 P 74e398a49767421291bb348c467b01c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:25:50.971386 30826 ts_tablet_manager.cc:1905] T 1dbf5b9330bb46999f0334a089a9ae56 P 74e398a49767421291bb348c467b01c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:25:50.981782 30822 tablet_replica.cc:330] T 1dbf5b9330bb46999f0334a089a9ae56 P 9b1c5d77b03843de81baaad1394ac7bf: stopping tablet replica
I20260810 03:25:50.982219 30822 raft_consensus.cc:2229] T 1dbf5b9330bb46999f0334a089a9ae56 P 9b1c5d77b03843de81baaad1394ac7bf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 03:25:50.982717 30822 raft_consensus.cc:2258] T 1dbf5b9330bb46999f0334a089a9ae56 P 9b1c5d77b03843de81baaad1394ac7bf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:25:50.982776 30821 tablet_replica.cc:330] T 1dbf5b9330bb46999f0334a089a9ae56 P 0d3f86f8815e4e28a37d1c6d9385da9e: stopping tablet replica
I20260810 03:25:50.982977 30821 raft_consensus.cc:2229] T 1dbf5b9330bb46999f0334a089a9ae56 P 0d3f86f8815e4e28a37d1c6d9385da9e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 03:25:50.983290 30821 raft_consensus.cc:2258] T 1dbf5b9330bb46999f0334a089a9ae56 P 0d3f86f8815e4e28a37d1c6d9385da9e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:25:50.984251 30822 ts_tablet_manager.cc:1905] T 1dbf5b9330bb46999f0334a089a9ae56 P 9b1c5d77b03843de81baaad1394ac7bf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:25:50.984452 30821 ts_tablet_manager.cc:1905] T 1dbf5b9330bb46999f0334a089a9ae56 P 0d3f86f8815e4e28a37d1c6d9385da9e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:25:50.991918 30822 ts_tablet_manager.cc:1918] T 1dbf5b9330bb46999f0334a089a9ae56 P 9b1c5d77b03843de81baaad1394ac7bf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 03:25:50.991918 30821 ts_tablet_manager.cc:1918] T 1dbf5b9330bb46999f0334a089a9ae56 P 0d3f86f8815e4e28a37d1c6d9385da9e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 03:25:50.992106 30821 log.cc:1196] T 1dbf5b9330bb46999f0334a089a9ae56 P 0d3f86f8815e4e28a37d1c6d9385da9e: Deleting WAL directory at /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786332336319544-28292-0/minicluster-data/ts-1/wal/wals/1dbf5b9330bb46999f0334a089a9ae56
I20260810 03:25:50.992913 30826 ts_tablet_manager.cc:1918] T 1dbf5b9330bb46999f0334a089a9ae56 P 74e398a49767421291bb348c467b01c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 03:25:50.993279 30826 log.cc:1196] T 1dbf5b9330bb46999f0334a089a9ae56 P 74e398a49767421291bb348c467b01c2: Deleting WAL directory at /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786332336319544-28292-0/minicluster-data/ts-0/wal/wals/1dbf5b9330bb46999f0334a089a9ae56
I20260810 03:25:50.994037 30826 ts_tablet_manager.cc:1939] T 1dbf5b9330bb46999f0334a089a9ae56 P 74e398a49767421291bb348c467b01c2: Deleting consensus metadata
I20260810 03:25:50.995848 29934 catalog_manager.cc:4841] TS 74e398a49767421291bb348c467b01c2 (127.27.161.1:34289): tablet 1dbf5b9330bb46999f0334a089a9ae56 (table test-table [id=c5935cd5fa7e4ebead92adc90c05a68b]) successfully deleted
I20260810 03:25:50.996316 30822 log.cc:1196] T 1dbf5b9330bb46999f0334a089a9ae56 P 9b1c5d77b03843de81baaad1394ac7bf: Deleting WAL directory at /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786332336319544-28292-0/minicluster-data/ts-2/wal/wals/1dbf5b9330bb46999f0334a089a9ae56
I20260810 03:25:50.996898 30822 ts_tablet_manager.cc:1939] T 1dbf5b9330bb46999f0334a089a9ae56 P 9b1c5d77b03843de81baaad1394ac7bf: Deleting consensus metadata
Aug 10 03:25:50 dist-test-slave-vpvm krb5kdc[29885](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332349, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.27.161.1@KRBTEST.COM
I20260810 03:25:50.998203 29934 catalog_manager.cc:4841] TS 9b1c5d77b03843de81baaad1394ac7bf (127.27.161.3:39751): tablet 1dbf5b9330bb46999f0334a089a9ae56 (table test-table [id=c5935cd5fa7e4ebead92adc90c05a68b]) successfully deleted
I20260810 03:25:51.002089 30821 ts_tablet_manager.cc:1939] T 1dbf5b9330bb46999f0334a089a9ae56 P 0d3f86f8815e4e28a37d1c6d9385da9e: Deleting consensus metadata
I20260810 03:25:51.003212 29933 catalog_manager.cc:4841] TS 0d3f86f8815e4e28a37d1c6d9385da9e (127.27.161.2:44877): tablet 1dbf5b9330bb46999f0334a089a9ae56 (table test-table [id=c5935cd5fa7e4ebead92adc90c05a68b]) successfully deleted
I20260810 03:25:51.006093 30796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:50.972862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35734 (local address 127.27.161.1:34289)
0810 03:25:50.973106 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0810 03:25:50.973113 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:25:50.985854 (+ 12741us) server_negotiation.cc:407] Connection header received
0810 03:25:50.986107 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:25:50.986113 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:25:50.986188 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:25:50.986276 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:25:50.988615 (+ 2339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:50.989823 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:50.991877 (+ 2054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:50.992688 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:50.992796 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:25:50.998297 (+ 5501us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:25:50.998317 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:25:50.998342 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:25:50.998395 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:25:51.002678 (+ 4283us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:25:51.003744 (+ 1066us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:25:51.003752 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:25:51.003754 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:25:51.003914 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:25:51.004323 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:25:51.004328 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:25:51.004331 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:25:51.004837 (+ 506us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:25:51.005192 (+ 355us) server_negotiation.cc:1035] Waiting for connection context
0810 03:25:51.005490 (+ 298us) server_negotiation.cc:299] Negotiation successful
0810 03:25:51.005732 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
W20260810 03:25:51.006913 30269 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35734
I20260810 03:25:51.025784 30833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:51.010568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56226 (local address 127.27.161.62:38165)
0810 03:25:51.011381 (+ 813us) server_negotiation.cc:206] Beginning negotiation
0810 03:25:51.011388 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:25:51.011406 (+ 18us) server_negotiation.cc:407] Connection header received
0810 03:25:51.011463 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:25:51.011467 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:25:51.011545 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:25:51.011641 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:25:51.013027 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:51.013895 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:51.015241 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:51.015824 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:51.015950 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:25:51.018841 (+ 2891us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:25:51.018848 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:25:51.018854 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:25:51.018909 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:25:51.022714 (+ 3805us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:25:51.023522 (+ 808us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:25:51.023528 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:25:51.023532 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:25:51.023682 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:25:51.024031 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:25:51.024035 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:25:51.024038 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:25:51.024382 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:25:51.024652 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0810 03:25:51.025156 (+ 504us) server_negotiation.cc:299] Negotiation successful
0810 03:25:51.025396 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":679,"thread_start_us":498,"threads_started":1}
W20260810 03:25:51.026455 29951 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56226
I20260810 03:25:51.028920 28292 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iBH5/build/asan/bin/kudu pid 30061
I20260810 03:25:51.045193 30796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:51.029785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35744 (local address 127.27.161.1:34289)
0810 03:25:51.031319 (+ 1534us) server_negotiation.cc:206] Beginning negotiation
0810 03:25:51.031325 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:25:51.031346 (+ 21us) server_negotiation.cc:407] Connection header received
0810 03:25:51.031413 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:25:51.031418 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:25:51.031511 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:25:51.031615 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:25:51.032764 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:51.033901 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:51.035169 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:51.035831 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:51.035955 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:25:51.038386 (+ 2431us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:25:51.038392 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:25:51.038397 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:25:51.038440 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:25:51.042297 (+ 3857us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:25:51.043051 (+ 754us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:25:51.043056 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:25:51.043060 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:25:51.043250 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:25:51.043646 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:25:51.043651 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:25:51.043655 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:25:51.044066 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:25:51.044326 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0810 03:25:51.044588 (+ 262us) server_negotiation.cc:299] Negotiation successful
0810 03:25:51.044827 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1393}
W20260810 03:25:51.046053 30269 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35744
W20260810 03:25:51.046638 28292 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task24iBH5/build/asan/bin/kudu pid 30061: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260810 03:25:51.046870 28292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iBH5/build/asan/bin/kudu with pid 30061
I20260810 03:25:51.082436 28292 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iBH5/build/asan/bin/kudu pid 30282
Aug 10 03:25:51 dist-test-slave-vpvm krb5kdc[29885](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332349, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.27.161.2@KRBTEST.COM
I20260810 03:25:51.098256 30835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:51.083243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55678 (local address 127.27.161.2:44877)
0810 03:25:51.083764 (+ 521us) server_negotiation.cc:206] Beginning negotiation
0810 03:25:51.083769 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:25:51.083786 (+ 17us) server_negotiation.cc:407] Connection header received
0810 03:25:51.083846 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:25:51.083850 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:25:51.083914 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:25:51.084034 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:25:51.085347 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:51.086204 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:51.087511 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:51.087961 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:51.088101 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:25:51.092448 (+ 4347us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:25:51.092465 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:25:51.092484 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:25:51.092529 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:25:51.095331 (+ 2802us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:25:51.096041 (+ 710us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:25:51.096050 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:25:51.096052 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:25:51.096262 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:25:51.096645 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:25:51.096649 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:25:51.096651 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:25:51.097039 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:25:51.097349 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0810 03:25:51.097636 (+ 287us) server_negotiation.cc:299] Negotiation successful
0810 03:25:51.097882 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":411,"thread_start_us":270,"threads_started":1}
W20260810 03:25:51.098922 30480 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55678
W20260810 03:25:51.099577 28292 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task24iBH5/build/asan/bin/kudu pid 30282: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260810 03:25:51.099754 28292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iBH5/build/asan/bin/kudu with pid 30282
I20260810 03:25:51.129233 28292 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iBH5/build/asan/bin/kudu pid 30492
Aug 10 03:25:51 dist-test-slave-vpvm krb5kdc[29885](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332349, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.27.161.3@KRBTEST.COM
I20260810 03:25:51.145474 30836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:51.130042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45648 (local address 127.27.161.3:39751)
0810 03:25:51.130530 (+ 488us) server_negotiation.cc:206] Beginning negotiation
0810 03:25:51.130534 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:25:51.130551 (+ 17us) server_negotiation.cc:407] Connection header received
0810 03:25:51.130607 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:25:51.130612 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:25:51.130687 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:25:51.130782 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:25:51.132090 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:51.132846 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:51.134322 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:51.134765 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:51.134875 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:25:51.139088 (+ 4213us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:25:51.139107 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:25:51.139127 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:25:51.139167 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:25:51.142499 (+ 3332us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:25:51.143279 (+ 780us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:25:51.143289 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:25:51.143291 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:25:51.143445 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:25:51.143847 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:25:51.143851 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:25:51.143853 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:25:51.144253 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:25:51.144567 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0810 03:25:51.144884 (+ 317us) server_negotiation.cc:299] Negotiation successful
0810 03:25:51.145135 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":370,"thread_start_us":234,"threads_started":1}
W20260810 03:25:51.146275 30686 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45648
W20260810 03:25:51.146992 28292 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task24iBH5/build/asan/bin/kudu pid 30492: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260810 03:25:51.147146 28292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iBH5/build/asan/bin/kudu with pid 30492
I20260810 03:25:51.175225 28292 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iBH5/build/asan/bin/kudu pid 29902
I20260810 03:25:51.176426 30010 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:25:51.294792 28292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iBH5/build/asan/bin/kudu with pid 29902
2026-08-10T03:25:51Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (5248 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786332336319544-28292-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 03:25:51 dist-test-slave-vpvm krb5kdc[30840](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 03:25:51 dist-test-slave-vpvm krb5kdc[30840](info): set up 2 sockets
Aug 10 03:25:51 dist-test-slave-vpvm krb5kdc[30840](info): commencing operation
krb5kdc: starting...
W20260810 03:25:52.511392 28292 mini_kdc.cc:121] Time spent starting KDC: real 1.163s user 0.001s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786332336319544-28292-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786332336319544-28292-0/krb5kdc/test-user.keytab.
Aug 10 03:25:52 dist-test-slave-vpvm krb5kdc[30840](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332352, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T03:25:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T03:25:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.161.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.161.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.161.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.62.keytab.
Entry for principal kudu/127.27.161.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.62.keytab.
I20260810 03:25:52.748750 28292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iBH5/build/asan/bin/kudu
/tmp/dist-test-task24iBH5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786332336319544-28292-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786332336319544-28292-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786332336319544-28292-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786332336319544-28292-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.161.62:38085
--webserver_interface=127.27.161.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.62.keytab
--principal=kudu/127.27.161.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.161.20:43225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.161.62:38085
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786332336319544-28292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786332336319544-28292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786332336319544-28292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:25:53.100417 30859 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:25:53.100751 30859 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:25:53.100888 30859 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:25:53.108397 30859 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:25:53.108522 30859 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260810 03:25:53.108572 30859 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:25:53.108661 30859 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:25:53.108731 30859 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:25:53.108800 30859 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:25:53.119797 30859 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.161.20:43225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786332336319544-28292-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786332336319544-28292-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.161.62:38085
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.62.keytab
--principal=kudu/127.27.161.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.161.62:38085
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786332336319544-28292-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.161.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786332336319544-28292-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:25:53.121469 30859 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:25:53.123015 30859 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:25:53.135406 30869 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:25:53.135456 30872 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:25:53.135478 30875 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:25:53.136291 30859 server_base.cc:1034] running on GCE node
I20260810 03:25:53.137146 30859 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:25:53.138751 30859 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:25:53.139993 30859 hybrid_clock.cc:648] HybridClock initialized: now 1786332353139924 us; error 40 us; skew 500 ppm
I20260810 03:25:53.140373 30859 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:25:53 dist-test-slave-vpvm krb5kdc[30840](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332353, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.161.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:25:53.143736 30859 init.cc:377] Logged in from keytab as kudu/127.27.161.62@KRBTEST.COM (short username kudu)
I20260810 03:25:53.145504 30859 webserver.cc:460] Webserver started at http://127.27.161.62:45963/ using document root <none> and password file <none>
I20260810 03:25:53.146337 30859 fs_manager.cc:359] Metadata directory not provided
I20260810 03:25:53.146435 30859 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:25:53.146723 30859 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:25:53.149827 30859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786332336319544-28292-0/minicluster-data/master-0/data/instance:
uuid: "0733cfa08dc743d8a587b321ff9eba88"
format_stamp: "Formatted at 2026-08-10 03:25:53 on dist-test-slave-vpvm"
server_key: "f103daed29ab4b82ce7232fdb8454889"
server_key_iv: "33ec9dcdbbcbe62328f0b3b949781a13"
server_key_version: "encryptionkey@0"
I20260810 03:25:53.150678 30859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786332336319544-28292-0/minicluster-data/master-0/wal/instance:
uuid: "0733cfa08dc743d8a587b321ff9eba88"
format_stamp: "Formatted at 2026-08-10 03:25:53 on dist-test-slave-vpvm"
server_key: "f103daed29ab4b82ce7232fdb8454889"
server_key_iv: "33ec9dcdbbcbe62328f0b3b949781a13"
server_key_version: "encryptionkey@0"
I20260810 03:25:53.156513 30859 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260810 03:25:53.160458 30885 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:25:53.162067 30859 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 03:25:53.162261 30859 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786332336319544-28292-0/minicluster-data/master-0/data,/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786332336319544-28292-0/minicluster-data/master-0/wal
uuid: "0733cfa08dc743d8a587b321ff9eba88"
format_stamp: "Formatted at 2026-08-10 03:25:53 on dist-test-slave-vpvm"
server_key: "f103daed29ab4b82ce7232fdb8454889"
server_key_iv: "33ec9dcdbbcbe62328f0b3b949781a13"
server_key_version: "encryptionkey@0"
I20260810 03:25:53.162492 30859 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786332336319544-28292-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786332336319544-28292-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786332336319544-28292-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:25:53.191556 30859 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:25:53.195869 30859 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:25:53.196141 30859 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:25:53.219483 30859 rpc_server.cc:307] RPC server started. Bound to: 127.27.161.62:38085
I20260810 03:25:53.219449 30983 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.161.62:38085 every 8 connection(s)
I20260810 03:25:53.221011 30859 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786332336319544-28292-0/minicluster-data/master-0/data/info.pb
I20260810 03:25:53.223034 28292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iBH5/build/asan/bin/kudu as pid 30859
I20260810 03:25:53.223254 28292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786332336319544-28292-0/minicluster-data/master-0/wal/instance
I20260810 03:25:53.224015 28292 external_mini_cluster.cc:1428] Setting key db29f0c7038161a8e45818d7926f62a3
I20260810 03:25:53.225634 30985 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 10 03:25:53 dist-test-slave-vpvm krb5kdc[30840](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332352, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.161.62@KRBTEST.COM
I20260810 03:25:53.238765 30985 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0733cfa08dc743d8a587b321ff9eba88: Bootstrap starting.
I20260810 03:25:53.242792 30985 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0733cfa08dc743d8a587b321ff9eba88: Neither blocks nor log segments found. Creating new log.
I20260810 03:25:53.244029 30985 log.cc:824] T 00000000000000000000000000000000 P 0733cfa08dc743d8a587b321ff9eba88: Log is configured to *not* fsync() on all Append() calls
I20260810 03:25:53.247152 30989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:53.226286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55646 (local address 127.27.161.62:38085)
0810 03:25:53.227148 (+ 862us) server_negotiation.cc:206] Beginning negotiation
0810 03:25:53.227161 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 03:25:53.227204 (+ 43us) server_negotiation.cc:407] Connection header received
0810 03:25:53.228850 (+ 1646us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:25:53.228874 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:25:53.229305 (+ 431us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:25:53.229665 (+ 360us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:25:53.231183 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:53.232559 (+ 1376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:53.233817 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:53.234566 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:53.234746 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:25:53.239236 (+ 4490us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:25:53.239245 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:25:53.239262 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:25:53.239314 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:25:53.242525 (+ 3211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:25:53.243359 (+ 834us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:25:53.243367 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:25:53.243375 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:25:53.243561 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:25:53.243977 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:25:53.243981 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:25:53.243983 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:25:53.244669 (+ 686us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:25:53.245010 (+ 341us) server_negotiation.cc:1035] Waiting for connection context
0810 03:25:53.245538 (+ 528us) server_negotiation.cc:299] Negotiation successful
0810 03:25:53.246054 (+ 516us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":640,"thread_start_us":412,"threads_started":1}
I20260810 03:25:53.247941 30985 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0733cfa08dc743d8a587b321ff9eba88: No bootstrap required, opened a new log
I20260810 03:25:53.256127 30985 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0733cfa08dc743d8a587b321ff9eba88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0733cfa08dc743d8a587b321ff9eba88" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 38085 } }
I20260810 03:25:53.256562 30985 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0733cfa08dc743d8a587b321ff9eba88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:25:53.256696 30985 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0733cfa08dc743d8a587b321ff9eba88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0733cfa08dc743d8a587b321ff9eba88, State: Initialized, Role: FOLLOWER
I20260810 03:25:53.257314 30985 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0733cfa08dc743d8a587b321ff9eba88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0733cfa08dc743d8a587b321ff9eba88" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 38085 } }
I20260810 03:25:53.257542 30985 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0733cfa08dc743d8a587b321ff9eba88 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 03:25:53.257685 30985 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0733cfa08dc743d8a587b321ff9eba88 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 03:25:53.257867 30985 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0733cfa08dc743d8a587b321ff9eba88 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:25:53.260097 30985 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0733cfa08dc743d8a587b321ff9eba88 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0733cfa08dc743d8a587b321ff9eba88" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 38085 } }
I20260810 03:25:53.260613 30985 leader_election.cc:302] T 00000000000000000000000000000000 P 0733cfa08dc743d8a587b321ff9eba88 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0733cfa08dc743d8a587b321ff9eba88; no voters:
I20260810 03:25:53.261241 30985 leader_election.cc:288] T 00000000000000000000000000000000 P 0733cfa08dc743d8a587b321ff9eba88 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 03:25:53.261411 30992 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0733cfa08dc743d8a587b321ff9eba88 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:25:53.262526 30992 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0733cfa08dc743d8a587b321ff9eba88 [term 1 LEADER]: Becoming Leader. State: Replica: 0733cfa08dc743d8a587b321ff9eba88, State: Running, Role: LEADER
I20260810 03:25:53.263041 30992 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0733cfa08dc743d8a587b321ff9eba88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0733cfa08dc743d8a587b321ff9eba88" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 38085 } }
I20260810 03:25:53.263489 30985 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0733cfa08dc743d8a587b321ff9eba88 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:25:53.265758 30993 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0733cfa08dc743d8a587b321ff9eba88 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0733cfa08dc743d8a587b321ff9eba88" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0733cfa08dc743d8a587b321ff9eba88" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 38085 } } }
I20260810 03:25:53.265728 30994 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0733cfa08dc743d8a587b321ff9eba88 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0733cfa08dc743d8a587b321ff9eba88. Latest consensus state: current_term: 1 leader_uuid: "0733cfa08dc743d8a587b321ff9eba88" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0733cfa08dc743d8a587b321ff9eba88" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 38085 } } }
I20260810 03:25:53.266289 30994 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0733cfa08dc743d8a587b321ff9eba88 [sys.catalog]: This master's current role is: LEADER
I20260810 03:25:53.266289 30993 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0733cfa08dc743d8a587b321ff9eba88 [sys.catalog]: This master's current role is: LEADER
I20260810 03:25:53.267306 31001 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 03:25:53.273155 31001 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 03:25:53.282466 31001 catalog_manager.cc:1347] Generated new cluster ID: 581251a9e83f4e96abd9aa9475436f6d
I20260810 03:25:53.282615 31001 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 03:25:53.304711 31001 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 03:25:53.306156 31001 catalog_manager.cc:1504] Loading token signing keys...
I20260810 03:25:53.314896 31001 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0733cfa08dc743d8a587b321ff9eba88: Generated new TSK 0
I20260810 03:25:53.316150 31001 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.161.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.161.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.161.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.1.keytab.
Entry for principal kudu/127.27.161.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.1.keytab.
I20260810 03:25:53.371685 28292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iBH5/build/asan/bin/kudu
/tmp/dist-test-task24iBH5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786332336319544-28292-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786332336319544-28292-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786332336319544-28292-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786332336319544-28292-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.161.1:0
--local_ip_for_outbound_sockets=127.27.161.1
--webserver_interface=127.27.161.1
--webserver_port=0
--tserver_master_addrs=127.27.161.62:38085
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.1.keytab
--principal=kudu/127.27.161.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.161.20:43225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786332336319544-28292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786332336319544-28292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786332336319544-28292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:25:53.743700 31020 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:25:53.744031 31020 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:25:53.744156 31020 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:25:53.752058 31020 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:25:53.752208 31020 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:25:53.752300 31020 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260810 03:25:53.752354 31020 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 03:25:53.752425 31020 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.161.1
I20260810 03:25:53.762751 31020 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.161.20:43225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786332336319544-28292-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786332336319544-28292-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.1.keytab
--principal=kudu/127.27.161.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.161.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786332336319544-28292-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.161.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.27.161.62:38085
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.161.1
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786332336319544-28292-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:25:53.764257 31020 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:25:53.765725 31020 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:25:53.779697 31025 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:25:53.779800 31029 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:25:53.779635 31026 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:25:53.780752 31020 server_base.cc:1034] running on GCE node
I20260810 03:25:53.781535 31020 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:25:53.782694 31020 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:25:53.783936 31020 hybrid_clock.cc:648] HybridClock initialized: now 1786332353783881 us; error 45 us; skew 500 ppm
I20260810 03:25:53.784294 31020 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:25:53 dist-test-slave-vpvm krb5kdc[30840](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332353, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.161.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:25:53.787475 31020 init.cc:377] Logged in from keytab as kudu/127.27.161.1@KRBTEST.COM (short username kudu)
I20260810 03:25:53.789000 31020 webserver.cc:460] Webserver started at http://127.27.161.1:41253/ using document root <none> and password file <none>
I20260810 03:25:53.789711 31020 fs_manager.cc:359] Metadata directory not provided
I20260810 03:25:53.789834 31020 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:25:53.790199 31020 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:25:53.793242 31020 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786332336319544-28292-0/minicluster-data/ts-0/data/instance:
uuid: "2831154de99842959a694f57c980b448"
format_stamp: "Formatted at 2026-08-10 03:25:53 on dist-test-slave-vpvm"
server_key: "b72ed7ce8405b26d11f62b265edc3296"
server_key_iv: "43a90568cabc30f4d19ab01d8de61d8c"
server_key_version: "encryptionkey@0"
I20260810 03:25:53.794060 31020 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786332336319544-28292-0/minicluster-data/ts-0/wal/instance:
uuid: "2831154de99842959a694f57c980b448"
format_stamp: "Formatted at 2026-08-10 03:25:53 on dist-test-slave-vpvm"
server_key: "b72ed7ce8405b26d11f62b265edc3296"
server_key_iv: "43a90568cabc30f4d19ab01d8de61d8c"
server_key_version: "encryptionkey@0"
I20260810 03:25:53.799703 31020 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260810 03:25:53.803903 31037 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:25:53.805474 31020 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260810 03:25:53.805655 31020 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786332336319544-28292-0/minicluster-data/ts-0/data,/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786332336319544-28292-0/minicluster-data/ts-0/wal
uuid: "2831154de99842959a694f57c980b448"
format_stamp: "Formatted at 2026-08-10 03:25:53 on dist-test-slave-vpvm"
server_key: "b72ed7ce8405b26d11f62b265edc3296"
server_key_iv: "43a90568cabc30f4d19ab01d8de61d8c"
server_key_version: "encryptionkey@0"
I20260810 03:25:53.805862 31020 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786332336319544-28292-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786332336319544-28292-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786332336319544-28292-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:25:53.830230 31020 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:25:53.834576 31020 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:25:53.834861 31020 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:25:53.837793 31020 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:25:53.837924 31020 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:25:53.838068 31020 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:25:53.838155 31020 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 03:25:53 dist-test-slave-vpvm krb5kdc[30840](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332353, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.161.1@KRBTEST.COM for kudu/127.27.161.62@KRBTEST.COM
I20260810 03:25:53.862098 31054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:53.838697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.161.1:37689 (local address 127.27.161.62:38085)
0810 03:25:53.839161 (+ 464us) server_negotiation.cc:206] Beginning negotiation
0810 03:25:53.839166 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:25:53.840461 (+ 1295us) server_negotiation.cc:407] Connection header received
0810 03:25:53.842616 (+ 2155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:25:53.842623 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:25:53.842713 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:25:53.842838 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:25:53.845110 (+ 2272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:53.846013 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:53.848509 (+ 2496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:53.848946 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:53.849078 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:25:53.854344 (+ 5266us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:25:53.854352 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:25:53.854357 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:25:53.854407 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:25:53.857753 (+ 3346us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:25:53.858979 (+ 1226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:25:53.858984 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:25:53.858987 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:25:53.859143 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:25:53.859872 (+ 729us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:25:53.859878 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:25:53.859881 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:25:53.860212 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:25:53.860358 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0810 03:25:53.861407 (+ 1049us) server_negotiation.cc:299] Negotiation successful
0810 03:25:53.861677 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":354,"thread_start_us":251,"threads_started":1}
I20260810 03:25:53.863345 31055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:53.839447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:38085 (local address 127.27.161.1:37689)
0810 03:25:53.840297 (+ 850us) negotiation.cc:107] Waiting for socket to connect
0810 03:25:53.840343 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0810 03:25:53.842306 (+ 1963us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:25:53.843138 (+ 832us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:25:53.843151 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:25:53.843984 (+ 833us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:25:53.844888 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:25:53.844919 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:25:53.846210 (+ 1291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:25:53.846215 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:25:53.848360 (+ 2145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:25:53.848376 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:25:53.849260 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:25:53.849271 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:25:53.849622 (+ 351us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:25:53.850529 (+ 907us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:25:53.850568 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:25:53.853926 (+ 3358us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:25:53.857933 (+ 4007us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:25:53.857946 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:25:53.858001 (+ 55us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:25:53.858779 (+ 778us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:25:53.859295 (+ 516us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:25:53.859301 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:25:53.859306 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:25:53.859696 (+ 390us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:25:53.860358 (+ 662us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:25:53.860368 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:25:53.860889 (+ 521us) client_negotiation.cc:769] Sending connection context
0810 03:25:53.861256 (+ 367us) client_negotiation.cc:240] Negotiation successful
0810 03:25:53.861870 (+ 614us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":580,"thread_start_us":355,"threads_started":1}
I20260810 03:25:53.885430 31172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:53.870842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:38085 (local address 127.27.161.1:32891)
0810 03:25:53.871485 (+ 643us) negotiation.cc:107] Waiting for socket to connect
0810 03:25:53.871505 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 03:25:53.871655 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:25:53.872044 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:25:53.872047 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:25:53.872525 (+ 478us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:25:53.872832 (+ 307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:25:53.872842 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:25:53.874341 (+ 1499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:25:53.874345 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:25:53.875399 (+ 1054us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:25:53.875413 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:25:53.876416 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:25:53.876420 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:25:53.876548 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:25:53.877199 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:25:53.877223 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:25:53.878460 (+ 1237us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:25:53.882278 (+ 3818us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:25:53.882282 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:25:53.882285 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:25:53.882709 (+ 424us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:25:53.883129 (+ 420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:25:53.883134 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:25:53.883145 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:25:53.883334 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:25:53.884028 (+ 694us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:25:53.884037 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:25:53.884251 (+ 214us) client_negotiation.cc:769] Sending connection context
0810 03:25:53.884401 (+ 150us) client_negotiation.cc:240] Negotiation successful
0810 03:25:53.884764 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":488,"thread_start_us":336,"threads_started":1}
I20260810 03:25:53.886010 31054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:53.870773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.161.1:32891 (local address 127.27.161.62:38085)
0810 03:25:53.871008 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0810 03:25:53.871015 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:25:53.871587 (+ 572us) server_negotiation.cc:407] Connection header received
0810 03:25:53.871803 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:25:53.871809 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:25:53.871917 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:25:53.872026 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:25:53.872975 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:53.874189 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:53.875529 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:53.876240 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:53.876369 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:25:53.878709 (+ 2340us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:25:53.878717 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:25:53.878723 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:25:53.878772 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:25:53.882124 (+ 3352us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:25:53.882821 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:25:53.882828 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:25:53.882832 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:25:53.883001 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:25:53.883500 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:25:53.883504 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:25:53.883507 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:25:53.883865 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:25:53.885377 (+ 1512us) server_negotiation.cc:1035] Waiting for connection context
0810 03:25:53.885490 (+ 113us) server_negotiation.cc:299] Negotiation successful
0810 03:25:53.885677 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260810 03:25:53.890105 31020 rpc_server.cc:307] RPC server started. Bound to: 127.27.161.1:33653
I20260810 03:25:53.890172 31225 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.161.1:33653 every 8 connection(s)
I20260810 03:25:53.891384 31020 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786332336319544-28292-0/minicluster-data/ts-0/data/info.pb
I20260810 03:25:53.893424 31226 heartbeater.cc:347] Connected to a master server at 127.27.161.62:38085
I20260810 03:25:53.893673 31226 heartbeater.cc:464] Registering TS with master...
I20260810 03:25:53.894392 31226 heartbeater.cc:511] Master 127.27.161.62:38085 requested a full tablet report, sending...
I20260810 03:25:53.896512 30908 ts_manager.cc:194] Registered new tserver with Master: 2831154de99842959a694f57c980b448 (127.27.161.1:33653)
I20260810 03:25:53.897078 28292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iBH5/build/asan/bin/kudu as pid 31020
I20260810 03:25:53.897267 28292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786332336319544-28292-0/minicluster-data/ts-0/wal/instance
I20260810 03:25:53.897895 28292 external_mini_cluster.cc:1428] Setting key 9d04fde4ae2f98473bdc010c74f618bc
I20260810 03:25:53.898137 30908 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.161.1@KRBTEST.COM'} at 127.27.161.1:37689
WARNING: no policy specified for kudu/127.27.161.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.161.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.161.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.2.keytab.
Entry for principal kudu/127.27.161.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.2.keytab.
I20260810 03:25:53.946842 28292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iBH5/build/asan/bin/kudu
/tmp/dist-test-task24iBH5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786332336319544-28292-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786332336319544-28292-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786332336319544-28292-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786332336319544-28292-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.161.2:0
--local_ip_for_outbound_sockets=127.27.161.2
--webserver_interface=127.27.161.2
--webserver_port=0
--tserver_master_addrs=127.27.161.62:38085
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.2.keytab
--principal=kudu/127.27.161.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.161.20:43225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786332336319544-28292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786332336319544-28292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786332336319544-28292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:25:54.305270 31231 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:25:54.305591 31231 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:25:54.305729 31231 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:25:54.313445 31231 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:25:54.313587 31231 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:25:54.313719 31231 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260810 03:25:54.313783 31231 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 03:25:54.313889 31231 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.161.2
I20260810 03:25:54.324543 31231 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.161.20:43225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786332336319544-28292-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786332336319544-28292-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.2.keytab
--principal=kudu/127.27.161.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.161.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786332336319544-28292-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.161.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.27.161.62:38085
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.161.2
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786332336319544-28292-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:25:54.325981 31231 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:25:54.327418 31231 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:25:54.342344 31238 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:25:54.342288 31240 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:25:54.342273 31242 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:25:54.342725 31231 server_base.cc:1034] running on GCE node
I20260810 03:25:54.343811 31231 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:25:54.344893 31231 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:25:54.346088 31231 hybrid_clock.cc:648] HybridClock initialized: now 1786332354346018 us; error 58 us; skew 500 ppm
I20260810 03:25:54.346459 31231 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:25:54 dist-test-slave-vpvm krb5kdc[30840](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332354, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.161.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:25:54.352980 31231 init.cc:377] Logged in from keytab as kudu/127.27.161.2@KRBTEST.COM (short username kudu)
I20260810 03:25:54.354929 31231 webserver.cc:460] Webserver started at http://127.27.161.2:38255/ using document root <none> and password file <none>
I20260810 03:25:54.355751 31231 fs_manager.cc:359] Metadata directory not provided
I20260810 03:25:54.355888 31231 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:25:54.356226 31231 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:25:54.359580 31231 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786332336319544-28292-0/minicluster-data/ts-1/data/instance:
uuid: "d9f56db87b6542bfaa31d2a10412e4be"
format_stamp: "Formatted at 2026-08-10 03:25:54 on dist-test-slave-vpvm"
server_key: "14651d900807922d82a62bf6e1ffcc2a"
server_key_iv: "105b2fab1a5059604b1a83c5701a55d3"
server_key_version: "encryptionkey@0"
I20260810 03:25:54.360409 31231 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786332336319544-28292-0/minicluster-data/ts-1/wal/instance:
uuid: "d9f56db87b6542bfaa31d2a10412e4be"
format_stamp: "Formatted at 2026-08-10 03:25:54 on dist-test-slave-vpvm"
server_key: "14651d900807922d82a62bf6e1ffcc2a"
server_key_iv: "105b2fab1a5059604b1a83c5701a55d3"
server_key_version: "encryptionkey@0"
I20260810 03:25:54.366283 31231 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260810 03:25:54.370360 31251 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:25:54.371805 31231 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 03:25:54.371971 31231 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786332336319544-28292-0/minicluster-data/ts-1/data,/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786332336319544-28292-0/minicluster-data/ts-1/wal
uuid: "d9f56db87b6542bfaa31d2a10412e4be"
format_stamp: "Formatted at 2026-08-10 03:25:54 on dist-test-slave-vpvm"
server_key: "14651d900807922d82a62bf6e1ffcc2a"
server_key_iv: "105b2fab1a5059604b1a83c5701a55d3"
server_key_version: "encryptionkey@0"
I20260810 03:25:54.372187 31231 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786332336319544-28292-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786332336319544-28292-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786332336319544-28292-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:25:54.405355 31231 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:25:54.410146 31231 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:25:54.410442 31231 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:25:54.413847 31231 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:25:54.414005 31231 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:25:54.414152 31231 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:25:54.414239 31231 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 03:25:54 dist-test-slave-vpvm krb5kdc[30840](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332354, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.161.2@KRBTEST.COM for kudu/127.27.161.62@KRBTEST.COM
I20260810 03:25:54.439702 31263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:54.414713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.161.2:45855 (local address 127.27.161.62:38085)
0810 03:25:54.415176 (+ 463us) server_negotiation.cc:206] Beginning negotiation
0810 03:25:54.415183 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:25:54.416587 (+ 1404us) server_negotiation.cc:407] Connection header received
0810 03:25:54.419256 (+ 2669us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:25:54.419263 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:25:54.419364 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:25:54.419482 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:25:54.422225 (+ 2743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:54.423288 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:54.425944 (+ 2656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:54.426556 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:54.426694 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:25:54.431430 (+ 4736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:25:54.431436 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:25:54.431440 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:25:54.431480 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:25:54.435384 (+ 3904us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:25:54.436580 (+ 1196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:25:54.436586 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:25:54.436589 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:25:54.436769 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:25:54.437543 (+ 774us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:25:54.437549 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:25:54.437553 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:25:54.437865 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:25:54.438051 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0810 03:25:54.439178 (+ 1127us) server_negotiation.cc:299] Negotiation successful
0810 03:25:54.439390 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":336,"thread_start_us":226,"threads_started":1}
I20260810 03:25:54.441509 31266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:54.415454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:38085 (local address 127.27.161.2:45855)
0810 03:25:54.416364 (+ 910us) negotiation.cc:107] Waiting for socket to connect
0810 03:25:54.416417 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0810 03:25:54.418871 (+ 2454us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:25:54.419843 (+ 972us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:25:54.419855 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:25:54.420860 (+ 1005us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:25:54.421935 (+ 1075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:25:54.421998 (+ 63us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:25:54.423532 (+ 1534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:25:54.423536 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:25:54.425731 (+ 2195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:25:54.425754 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:25:54.426858 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:25:54.426872 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:25:54.427211 (+ 339us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:25:54.428056 (+ 845us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:25:54.428082 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:25:54.431124 (+ 3042us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:25:54.435606 (+ 4482us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:25:54.435619 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:25:54.435645 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:25:54.436393 (+ 748us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:25:54.436946 (+ 553us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:25:54.436952 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:25:54.436955 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:25:54.437377 (+ 422us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:25:54.438040 (+ 663us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:25:54.438051 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:25:54.438673 (+ 622us) client_negotiation.cc:769] Sending connection context
0810 03:25:54.439028 (+ 355us) client_negotiation.cc:240] Negotiation successful
0810 03:25:54.439714 (+ 686us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":596,"thread_start_us":368,"threads_started":1}
I20260810 03:25:54.465046 31263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:54.449128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.161.2:37083 (local address 127.27.161.62:38085)
0810 03:25:54.449344 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0810 03:25:54.449351 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:25:54.450363 (+ 1012us) server_negotiation.cc:407] Connection header received
0810 03:25:54.450576 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:25:54.450582 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:25:54.450677 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:25:54.450782 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:25:54.452472 (+ 1690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:54.453206 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:54.455537 (+ 2331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:54.455982 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:54.456105 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:25:54.458920 (+ 2815us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:25:54.458925 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:25:54.458928 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:25:54.458965 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:25:54.461494 (+ 2529us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:25:54.462393 (+ 899us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:25:54.462398 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:25:54.462400 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:25:54.462540 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:25:54.463043 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:25:54.463050 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:25:54.463053 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:25:54.463605 (+ 552us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:25:54.463718 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0810 03:25:54.464440 (+ 722us) server_negotiation.cc:299] Negotiation successful
0810 03:25:54.464690 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260810 03:25:54.465293 31377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:54.449128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:38085 (local address 127.27.161.2:37083)
0810 03:25:54.450229 (+ 1101us) negotiation.cc:107] Waiting for socket to connect
0810 03:25:54.450255 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0810 03:25:54.450432 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:25:54.450891 (+ 459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:25:54.450898 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:25:54.451727 (+ 829us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:25:54.452277 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:25:54.452304 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:25:54.453365 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:25:54.453371 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:25:54.455343 (+ 1972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:25:54.455360 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:25:54.456206 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:25:54.456217 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:25:54.456408 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:25:54.457212 (+ 804us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:25:54.457241 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:25:54.458687 (+ 1446us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:25:54.461630 (+ 2943us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:25:54.461634 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:25:54.461639 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:25:54.462271 (+ 632us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:25:54.462680 (+ 409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:25:54.462684 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:25:54.462687 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:25:54.462881 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:25:54.463805 (+ 924us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:25:54.463810 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:25:54.464061 (+ 251us) client_negotiation.cc:769] Sending connection context
0810 03:25:54.464314 (+ 253us) client_negotiation.cc:240] Negotiation successful
0810 03:25:54.464717 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":921,"thread_start_us":726,"threads_started":1}
I20260810 03:25:54.468744 31231 rpc_server.cc:307] RPC server started. Bound to: 127.27.161.2:33631
I20260810 03:25:54.468773 31447 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.161.2:33631 every 8 connection(s)
I20260810 03:25:54.470105 31231 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786332336319544-28292-0/minicluster-data/ts-1/data/info.pb
I20260810 03:25:54.472173 28292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iBH5/build/asan/bin/kudu as pid 31231
I20260810 03:25:54.472182 31450 heartbeater.cc:347] Connected to a master server at 127.27.161.62:38085
I20260810 03:25:54.472337 28292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786332336319544-28292-0/minicluster-data/ts-1/wal/instance
I20260810 03:25:54.472434 31450 heartbeater.cc:464] Registering TS with master...
I20260810 03:25:54.472820 28292 external_mini_cluster.cc:1428] Setting key 3e4f37ba222db807a88c01dccbd5e600
I20260810 03:25:54.473193 31450 heartbeater.cc:511] Master 127.27.161.62:38085 requested a full tablet report, sending...
I20260810 03:25:54.474910 30906 ts_manager.cc:194] Registered new tserver with Master: d9f56db87b6542bfaa31d2a10412e4be (127.27.161.2:33631)
I20260810 03:25:54.475839 30906 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.161.2@KRBTEST.COM'} at 127.27.161.2:45855
WARNING: no policy specified for kudu/127.27.161.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.161.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.161.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.3.keytab.
Entry for principal kudu/127.27.161.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.3.keytab.
I20260810 03:25:54.522564 28292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iBH5/build/asan/bin/kudu
/tmp/dist-test-task24iBH5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786332336319544-28292-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786332336319544-28292-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786332336319544-28292-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786332336319544-28292-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.161.3:0
--local_ip_for_outbound_sockets=127.27.161.3
--webserver_interface=127.27.161.3
--webserver_port=0
--tserver_master_addrs=127.27.161.62:38085
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.3.keytab
--principal=kudu/127.27.161.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.161.20:43225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786332336319544-28292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786332336319544-28292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786332336319544-28292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:25:54.866745 31455 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:25:54.867039 31455 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:25:54.867161 31455 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:25:54.875241 31455 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:25:54.875375 31455 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:25:54.875497 31455 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260810 03:25:54.875563 31455 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 03:25:54.875638 31455 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.161.3
I20260810 03:25:54.885864 31455 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.161.20:43225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786332336319544-28292-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786332336319544-28292-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.3.keytab
--principal=kudu/127.27.161.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.161.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786332336319544-28292-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.161.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.27.161.62:38085
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.161.3
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786332336319544-28292-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:25:54.887274 31455 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:25:54.888696 31455 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 03:25:54.901885 31226 heartbeater.cc:503] Master 127.27.161.62:38085 was elected leader, sending a full tablet report...
W20260810 03:25:54.902474 31465 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:25:54.902711 31464 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:25:54.903327 31467 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:25:54.904109 31455 server_base.cc:1034] running on GCE node
I20260810 03:25:54.904803 31455 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:25:54.905869 31455 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:25:54.907094 31455 hybrid_clock.cc:648] HybridClock initialized: now 1786332354907009 us; error 75 us; skew 500 ppm
I20260810 03:25:54.907441 31455 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:25:54 dist-test-slave-vpvm krb5kdc[30840](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332354, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.161.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:25:54.910905 31455 init.cc:377] Logged in from keytab as kudu/127.27.161.3@KRBTEST.COM (short username kudu)
I20260810 03:25:54.912624 31455 webserver.cc:460] Webserver started at http://127.27.161.3:38635/ using document root <none> and password file <none>
I20260810 03:25:54.913372 31455 fs_manager.cc:359] Metadata directory not provided
I20260810 03:25:54.913498 31455 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:25:54.913823 31455 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:25:54.917062 31455 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786332336319544-28292-0/minicluster-data/ts-2/data/instance:
uuid: "a16f2984931649a2a594bdd90b4debae"
format_stamp: "Formatted at 2026-08-10 03:25:54 on dist-test-slave-vpvm"
server_key: "b12ce6078b110106a520be6b8f5d3d67"
server_key_iv: "bc7597983b22e10694f37414f6cf8e91"
server_key_version: "encryptionkey@0"
I20260810 03:25:54.917903 31455 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786332336319544-28292-0/minicluster-data/ts-2/wal/instance:
uuid: "a16f2984931649a2a594bdd90b4debae"
format_stamp: "Formatted at 2026-08-10 03:25:54 on dist-test-slave-vpvm"
server_key: "b12ce6078b110106a520be6b8f5d3d67"
server_key_iv: "bc7597983b22e10694f37414f6cf8e91"
server_key_version: "encryptionkey@0"
I20260810 03:25:54.923707 31455 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260810 03:25:54.927690 31478 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:25:54.929194 31455 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260810 03:25:54.929371 31455 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786332336319544-28292-0/minicluster-data/ts-2/data,/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786332336319544-28292-0/minicluster-data/ts-2/wal
uuid: "a16f2984931649a2a594bdd90b4debae"
format_stamp: "Formatted at 2026-08-10 03:25:54 on dist-test-slave-vpvm"
server_key: "b12ce6078b110106a520be6b8f5d3d67"
server_key_iv: "bc7597983b22e10694f37414f6cf8e91"
server_key_version: "encryptionkey@0"
I20260810 03:25:54.929705 31455 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786332336319544-28292-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786332336319544-28292-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786332336319544-28292-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:25:54.949730 31455 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:25:54.954468 31455 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:25:54.954763 31455 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:25:54.958099 31455 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:25:54.958228 31455 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:25:54.958353 31455 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:25:54.958438 31455 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 03:25:54 dist-test-slave-vpvm krb5kdc[30840](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332354, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.161.3@KRBTEST.COM for kudu/127.27.161.62@KRBTEST.COM
I20260810 03:25:54.981354 31263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:54.958839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.161.3:37261 (local address 127.27.161.62:38085)
0810 03:25:54.959040 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0810 03:25:54.959047 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:25:54.960532 (+ 1485us) server_negotiation.cc:407] Connection header received
0810 03:25:54.962763 (+ 2231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:25:54.962770 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:25:54.962862 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:25:54.962951 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:25:54.965214 (+ 2263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:54.966182 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:54.968071 (+ 1889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:54.968810 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:54.968963 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:25:54.974020 (+ 5057us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:25:54.974029 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:25:54.974033 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:25:54.974086 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:25:54.977304 (+ 3218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:25:54.978536 (+ 1232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:25:54.978544 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:25:54.978547 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:25:54.978713 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:25:54.979345 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:25:54.979349 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:25:54.979351 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:25:54.979757 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:25:54.979915 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0810 03:25:54.980738 (+ 823us) server_negotiation.cc:299] Negotiation successful
0810 03:25:54.980981 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260810 03:25:54.982756 31492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:54.959489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:38085 (local address 127.27.161.3:37261)
0810 03:25:54.960360 (+ 871us) negotiation.cc:107] Waiting for socket to connect
0810 03:25:54.960409 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0810 03:25:54.962448 (+ 2039us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:25:54.963274 (+ 826us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:25:54.963287 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:25:54.964163 (+ 876us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:25:54.965019 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:25:54.965038 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:25:54.966383 (+ 1345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:25:54.966388 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:25:54.967895 (+ 1507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:25:54.967928 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:25:54.969113 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:25:54.969123 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:25:54.969338 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:25:54.970080 (+ 742us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:25:54.970109 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:25:54.973637 (+ 3528us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:25:54.977514 (+ 3877us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:25:54.977529 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:25:54.977552 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:25:54.978353 (+ 801us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:25:54.978845 (+ 492us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:25:54.978852 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:25:54.978859 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:25:54.979212 (+ 353us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:25:54.979900 (+ 688us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:25:54.979909 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:25:54.980335 (+ 426us) client_negotiation.cc:769] Sending connection context
0810 03:25:54.980600 (+ 265us) client_negotiation.cc:240] Negotiation successful
0810 03:25:54.981196 (+ 596us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":614,"thread_start_us":378,"threads_started":1}
I20260810 03:25:55.003815 31263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:54.988747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.161.3:54941 (local address 127.27.161.62:38085)
0810 03:25:54.988952 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0810 03:25:54.988960 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 03:25:54.989567 (+ 607us) server_negotiation.cc:407] Connection header received
0810 03:25:54.989783 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:25:54.989788 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:25:54.989853 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:25:54.989936 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:25:54.991483 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:54.992281 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:54.994568 (+ 2287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:54.995087 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:54.995214 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:25:54.997265 (+ 2051us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:25:54.997271 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:25:54.997274 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:25:54.997311 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:25:55.000536 (+ 3225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:25:55.001470 (+ 934us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:25:55.001476 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:25:55.001480 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:25:55.001691 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:25:55.002154 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:25:55.002160 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:25:55.002163 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:25:55.002562 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:25:55.002725 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0810 03:25:55.003174 (+ 449us) server_negotiation.cc:299] Negotiation successful
0810 03:25:55.003421 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260810 03:25:55.003882 31553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:54.988723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:38085 (local address 127.27.161.3:54941)
0810 03:25:54.989440 (+ 717us) negotiation.cc:107] Waiting for socket to connect
0810 03:25:54.989463 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0810 03:25:54.989640 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:25:54.990007 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:25:54.990025 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:25:54.990758 (+ 733us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:25:54.991291 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:25:54.991310 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:25:54.992438 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:25:54.992444 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:25:54.994359 (+ 1915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:25:54.994378 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:25:54.995366 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:25:54.995377 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:25:54.995549 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:25:54.996236 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:25:54.996254 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:25:54.997061 (+ 807us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:25:55.000682 (+ 3621us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:25:55.000688 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:25:55.000695 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:25:55.001312 (+ 617us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:25:55.001834 (+ 522us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:25:55.001840 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:25:55.001847 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:25:55.002037 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:25:55.002691 (+ 654us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:25:55.002696 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:25:55.002898 (+ 202us) client_negotiation.cc:769] Sending connection context
0810 03:25:55.003028 (+ 130us) client_negotiation.cc:240] Negotiation successful
0810 03:25:55.003421 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":543,"thread_start_us":342,"threads_started":1}
I20260810 03:25:55.016316 31455 rpc_server.cc:307] RPC server started. Bound to: 127.27.161.3:42247
I20260810 03:25:55.016351 31660 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.161.3:42247 every 8 connection(s)
I20260810 03:25:55.017660 31455 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786332336319544-28292-0/minicluster-data/ts-2/data/info.pb
I20260810 03:25:55.020193 31661 heartbeater.cc:347] Connected to a master server at 127.27.161.62:38085
I20260810 03:25:55.020443 31661 heartbeater.cc:464] Registering TS with master...
I20260810 03:25:55.021135 31661 heartbeater.cc:511] Master 127.27.161.62:38085 requested a full tablet report, sending...
I20260810 03:25:55.022733 30906 ts_manager.cc:194] Registered new tserver with Master: a16f2984931649a2a594bdd90b4debae (127.27.161.3:42247)
I20260810 03:25:55.023603 30906 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.161.3@KRBTEST.COM'} at 127.27.161.3:37261
I20260810 03:25:55.027498 28292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iBH5/build/asan/bin/kudu as pid 31455
I20260810 03:25:55.027666 28292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786332336319544-28292-0/minicluster-data/ts-2/wal/instance
I20260810 03:25:55.028084 28292 external_mini_cluster.cc:1428] Setting key 9b06cc2da13b2b2c8f0a9441a577174d
I20260810 03:25:55.031529 28292 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 10 03:25:55 dist-test-slave-vpvm krb5kdc[30840](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332355, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 10 03:25:55 dist-test-slave-vpvm krb5kdc[30840](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332355, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.161.62@KRBTEST.COM
I20260810 03:25:55.089430 31263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:55.074940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55662 (local address 127.27.161.62:38085)
0810 03:25:55.075180 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0810 03:25:55.075187 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:25:55.075396 (+ 209us) server_negotiation.cc:407] Connection header received
0810 03:25:55.075575 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:25:55.075580 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:25:55.075656 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:25:55.075745 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:25:55.077301 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:55.078132 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:55.079403 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:55.079824 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:55.079953 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:25:55.084043 (+ 4090us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:25:55.084049 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:25:55.084052 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:25:55.084097 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:25:55.086741 (+ 2644us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:25:55.087426 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:25:55.087431 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:25:55.087433 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:25:55.087578 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:25:55.087972 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:25:55.087976 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:25:55.087979 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:25:55.088302 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:25:55.088568 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0810 03:25:55.088865 (+ 297us) server_negotiation.cc:299] Negotiation successful
0810 03:25:55.089084 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260810 03:25:55.093866 30906 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55662:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 03:25:55.100986 30906 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 03:25:55.126416 31678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:55.114220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.3:42247 (local address 127.0.0.1:44192)
0810 03:25:55.116604 (+ 2384us) negotiation.cc:107] Waiting for socket to connect
0810 03:25:55.116651 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0810 03:25:55.116874 (+ 223us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:25:55.118180 (+ 1306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:25:55.118187 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:25:55.118221 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:25:55.118646 (+ 425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:25:55.118660 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:25:55.120835 (+ 2175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:25:55.120840 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:25:55.123404 (+ 2564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:25:55.123417 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:25:55.125266 (+ 1849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:25:55.125273 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:25:55.125470 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:25:55.125525 (+ 55us) client_negotiation.cc:769] Sending connection context
0810 03:25:55.125675 (+ 150us) client_negotiation.cc:240] Negotiation successful
0810 03:25:55.125890 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2083,"mutex_wait_us":304,"thread_start_us":3074,"threads_started":1}
I20260810 03:25:55.127107 31683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:55.118306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40112 (local address 127.27.161.2:33631)
0810 03:25:55.119185 (+ 879us) server_negotiation.cc:206] Beginning negotiation
0810 03:25:55.119194 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 03:25:55.119717 (+ 523us) server_negotiation.cc:407] Connection header received
0810 03:25:55.119851 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:25:55.119857 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:25:55.120019 (+ 162us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:25:55.120181 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:25:55.120761 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:55.122137 (+ 1376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:55.124473 (+ 2336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:55.125617 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:55.125756 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:25:55.125945 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0810 03:25:55.126700 (+ 755us) server_negotiation.cc:299] Negotiation successful
0810 03:25:55.126886 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":716,"thread_start_us":593,"threads_started":1}
I20260810 03:25:55.127209 31680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:55.114978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59472 (local address 127.27.161.1:33653)
0810 03:25:55.117320 (+ 2342us) server_negotiation.cc:206] Beginning negotiation
0810 03:25:55.117329 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 03:25:55.117874 (+ 545us) server_negotiation.cc:407] Connection header received
0810 03:25:55.118004 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:25:55.118011 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:25:55.118236 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:25:55.118423 (+ 187us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:25:55.119376 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:55.120841 (+ 1465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:55.123813 (+ 2972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:55.125244 (+ 1431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:55.125370 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:25:55.125554 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0810 03:25:55.126731 (+ 1177us) server_negotiation.cc:299] Negotiation successful
0810 03:25:55.126913 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2189,"thread_start_us":2058,"threads_started":1}
I20260810 03:25:55.127672 31682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:55.117759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.2:33631 (local address 127.0.0.1:40112)
0810 03:25:55.119597 (+ 1838us) negotiation.cc:107] Waiting for socket to connect
0810 03:25:55.119611 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 03:25:55.119721 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:25:55.120189 (+ 468us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:25:55.120193 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:25:55.120215 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:25:55.120604 (+ 389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:25:55.120613 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:25:55.122470 (+ 1857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:25:55.122476 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:25:55.124325 (+ 1849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:25:55.124337 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:25:55.126106 (+ 1769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:25:55.126114 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:25:55.126304 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:25:55.126336 (+ 32us) client_negotiation.cc:769] Sending connection context
0810 03:25:55.126429 (+ 93us) client_negotiation.cc:240] Negotiation successful
0810 03:25:55.126562 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1745,"mutex_wait_us":4,"thread_start_us":1602,"threads_started":1}
I20260810 03:25:55.126582 31677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:55.114380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.1:33653 (local address 127.0.0.1:59472)
0810 03:25:55.117595 (+ 3215us) negotiation.cc:107] Waiting for socket to connect
0810 03:25:55.117612 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 03:25:55.117744 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:25:55.118644 (+ 900us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:25:55.118648 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:25:55.118673 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:25:55.119142 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:25:55.119156 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:25:55.120964 (+ 1808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:25:55.120968 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:25:55.123617 (+ 2649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:25:55.123655 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:25:55.125696 (+ 2041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:25:55.125702 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:25:55.125875 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:25:55.125906 (+ 31us) client_negotiation.cc:769] Sending connection context
0810 03:25:55.126027 (+ 121us) client_negotiation.cc:240] Negotiation successful
0810 03:25:55.126185 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3075,"mutex_wait_us":81,"thread_start_us":1774,"threads_started":1}
I20260810 03:25:55.128003 31679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:55.114702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44192 (local address 127.27.161.3:42247)
0810 03:25:55.115916 (+ 1214us) server_negotiation.cc:206] Beginning negotiation
0810 03:25:55.115927 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 03:25:55.117105 (+ 1178us) server_negotiation.cc:407] Connection header received
0810 03:25:55.117239 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:25:55.117249 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:25:55.117555 (+ 306us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:25:55.117762 (+ 207us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:25:55.118875 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:55.120640 (+ 1765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:55.123575 (+ 2935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:55.125058 (+ 1483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:55.125204 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:25:55.125471 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0810 03:25:55.127485 (+ 2014us) server_negotiation.cc:299] Negotiation successful
0810 03:25:55.127682 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1026,"thread_start_us":848,"threads_started":1}
I20260810 03:25:55.131297 31559 tablet_service.cc:1467] Processing CreateTablet for tablet 9c7a9e6a22314a258baa48a0ed5bf069 (DEFAULT_TABLE table=test-table [id=5cf9906ca78a4ac684e6520a2f83a069]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:25:55.131413 31331 tablet_service.cc:1467] Processing CreateTablet for tablet 9c7a9e6a22314a258baa48a0ed5bf069 (DEFAULT_TABLE table=test-table [id=5cf9906ca78a4ac684e6520a2f83a069]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:25:55.132594 31559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c7a9e6a22314a258baa48a0ed5bf069. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:25:55.132827 31331 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c7a9e6a22314a258baa48a0ed5bf069. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:25:55.132629 31126 tablet_service.cc:1467] Processing CreateTablet for tablet 9c7a9e6a22314a258baa48a0ed5bf069 (DEFAULT_TABLE table=test-table [id=5cf9906ca78a4ac684e6520a2f83a069]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:25:55.135111 31126 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c7a9e6a22314a258baa48a0ed5bf069. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:25:55.144799 31684 tablet_bootstrap.cc:492] T 9c7a9e6a22314a258baa48a0ed5bf069 P d9f56db87b6542bfaa31d2a10412e4be: Bootstrap starting.
I20260810 03:25:55.148718 31687 tablet_bootstrap.cc:492] T 9c7a9e6a22314a258baa48a0ed5bf069 P a16f2984931649a2a594bdd90b4debae: Bootstrap starting.
I20260810 03:25:55.149133 31684 tablet_bootstrap.cc:654] T 9c7a9e6a22314a258baa48a0ed5bf069 P d9f56db87b6542bfaa31d2a10412e4be: Neither blocks nor log segments found. Creating new log.
I20260810 03:25:55.150269 31688 tablet_bootstrap.cc:492] T 9c7a9e6a22314a258baa48a0ed5bf069 P 2831154de99842959a694f57c980b448: Bootstrap starting.
I20260810 03:25:55.150756 31684 log.cc:824] T 9c7a9e6a22314a258baa48a0ed5bf069 P d9f56db87b6542bfaa31d2a10412e4be: Log is configured to *not* fsync() on all Append() calls
I20260810 03:25:55.152302 31687 tablet_bootstrap.cc:654] T 9c7a9e6a22314a258baa48a0ed5bf069 P a16f2984931649a2a594bdd90b4debae: Neither blocks nor log segments found. Creating new log.
I20260810 03:25:55.153298 31688 tablet_bootstrap.cc:654] T 9c7a9e6a22314a258baa48a0ed5bf069 P 2831154de99842959a694f57c980b448: Neither blocks nor log segments found. Creating new log.
I20260810 03:25:55.153590 31687 log.cc:824] T 9c7a9e6a22314a258baa48a0ed5bf069 P a16f2984931649a2a594bdd90b4debae: Log is configured to *not* fsync() on all Append() calls
I20260810 03:25:55.154294 31684 tablet_bootstrap.cc:492] T 9c7a9e6a22314a258baa48a0ed5bf069 P d9f56db87b6542bfaa31d2a10412e4be: No bootstrap required, opened a new log
I20260810 03:25:55.154348 31688 log.cc:824] T 9c7a9e6a22314a258baa48a0ed5bf069 P 2831154de99842959a694f57c980b448: Log is configured to *not* fsync() on all Append() calls
I20260810 03:25:55.154589 31684 ts_tablet_manager.cc:1397] T 9c7a9e6a22314a258baa48a0ed5bf069 P d9f56db87b6542bfaa31d2a10412e4be: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260810 03:25:55.156726 31687 tablet_bootstrap.cc:492] T 9c7a9e6a22314a258baa48a0ed5bf069 P a16f2984931649a2a594bdd90b4debae: No bootstrap required, opened a new log
I20260810 03:25:55.157020 31687 ts_tablet_manager.cc:1397] T 9c7a9e6a22314a258baa48a0ed5bf069 P a16f2984931649a2a594bdd90b4debae: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260810 03:25:55.157357 31688 tablet_bootstrap.cc:492] T 9c7a9e6a22314a258baa48a0ed5bf069 P 2831154de99842959a694f57c980b448: No bootstrap required, opened a new log
I20260810 03:25:55.157632 31688 ts_tablet_manager.cc:1397] T 9c7a9e6a22314a258baa48a0ed5bf069 P 2831154de99842959a694f57c980b448: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.003s
I20260810 03:25:55.163059 31684 raft_consensus.cc:348] T 9c7a9e6a22314a258baa48a0ed5bf069 P d9f56db87b6542bfaa31d2a10412e4be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a16f2984931649a2a594bdd90b4debae" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 42247 } } peers { permanent_uuid: "2831154de99842959a694f57c980b448" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 33653 } } peers { permanent_uuid: "d9f56db87b6542bfaa31d2a10412e4be" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 33631 } }
I20260810 03:25:55.163511 31684 raft_consensus.cc:374] T 9c7a9e6a22314a258baa48a0ed5bf069 P d9f56db87b6542bfaa31d2a10412e4be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:25:55.163614 31684 raft_consensus.cc:729] T 9c7a9e6a22314a258baa48a0ed5bf069 P d9f56db87b6542bfaa31d2a10412e4be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9f56db87b6542bfaa31d2a10412e4be, State: Initialized, Role: FOLLOWER
I20260810 03:25:55.164330 31684 consensus_queue.cc:260] T 9c7a9e6a22314a258baa48a0ed5bf069 P d9f56db87b6542bfaa31d2a10412e4be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a16f2984931649a2a594bdd90b4debae" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 42247 } } peers { permanent_uuid: "2831154de99842959a694f57c980b448" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 33653 } } peers { permanent_uuid: "d9f56db87b6542bfaa31d2a10412e4be" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 33631 } }
I20260810 03:25:55.165458 31450 heartbeater.cc:503] Master 127.27.161.62:38085 was elected leader, sending a full tablet report...
I20260810 03:25:55.167882 31687 raft_consensus.cc:348] T 9c7a9e6a22314a258baa48a0ed5bf069 P a16f2984931649a2a594bdd90b4debae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a16f2984931649a2a594bdd90b4debae" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 42247 } } peers { permanent_uuid: "2831154de99842959a694f57c980b448" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 33653 } } peers { permanent_uuid: "d9f56db87b6542bfaa31d2a10412e4be" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 33631 } }
I20260810 03:25:55.168385 31687 raft_consensus.cc:374] T 9c7a9e6a22314a258baa48a0ed5bf069 P a16f2984931649a2a594bdd90b4debae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:25:55.168138 31688 raft_consensus.cc:348] T 9c7a9e6a22314a258baa48a0ed5bf069 P 2831154de99842959a694f57c980b448 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a16f2984931649a2a594bdd90b4debae" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 42247 } } peers { permanent_uuid: "2831154de99842959a694f57c980b448" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 33653 } } peers { permanent_uuid: "d9f56db87b6542bfaa31d2a10412e4be" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 33631 } }
I20260810 03:25:55.168535 31688 raft_consensus.cc:374] T 9c7a9e6a22314a258baa48a0ed5bf069 P 2831154de99842959a694f57c980b448 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:25:55.168538 31687 raft_consensus.cc:729] T 9c7a9e6a22314a258baa48a0ed5bf069 P a16f2984931649a2a594bdd90b4debae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a16f2984931649a2a594bdd90b4debae, State: Initialized, Role: FOLLOWER
I20260810 03:25:55.168632 31688 raft_consensus.cc:729] T 9c7a9e6a22314a258baa48a0ed5bf069 P 2831154de99842959a694f57c980b448 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2831154de99842959a694f57c980b448, State: Initialized, Role: FOLLOWER
I20260810 03:25:55.169196 31688 consensus_queue.cc:260] T 9c7a9e6a22314a258baa48a0ed5bf069 P 2831154de99842959a694f57c980b448 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a16f2984931649a2a594bdd90b4debae" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 42247 } } peers { permanent_uuid: "2831154de99842959a694f57c980b448" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 33653 } } peers { permanent_uuid: "d9f56db87b6542bfaa31d2a10412e4be" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 33631 } }
I20260810 03:25:55.169196 31687 consensus_queue.cc:260] T 9c7a9e6a22314a258baa48a0ed5bf069 P a16f2984931649a2a594bdd90b4debae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a16f2984931649a2a594bdd90b4debae" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 42247 } } peers { permanent_uuid: "2831154de99842959a694f57c980b448" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 33653 } } peers { permanent_uuid: "d9f56db87b6542bfaa31d2a10412e4be" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 33631 } }
I20260810 03:25:55.170666 31684 ts_tablet_manager.cc:1428] T 9c7a9e6a22314a258baa48a0ed5bf069 P d9f56db87b6542bfaa31d2a10412e4be: Time spent starting tablet: real 0.016s user 0.011s sys 0.000s
I20260810 03:25:55.172503 31688 ts_tablet_manager.cc:1428] T 9c7a9e6a22314a258baa48a0ed5bf069 P 2831154de99842959a694f57c980b448: Time spent starting tablet: real 0.015s user 0.011s sys 0.004s
I20260810 03:25:55.175062 31661 heartbeater.cc:503] Master 127.27.161.62:38085 was elected leader, sending a full tablet report...
I20260810 03:25:55.175733 31687 ts_tablet_manager.cc:1428] T 9c7a9e6a22314a258baa48a0ed5bf069 P a16f2984931649a2a594bdd90b4debae: Time spent starting tablet: real 0.019s user 0.019s sys 0.000s
W20260810 03:25:55.222425 31451 tablet.cc:2369] T 9c7a9e6a22314a258baa48a0ed5bf069 P d9f56db87b6542bfaa31d2a10412e4be: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 03:25:55.269451 31662 tablet.cc:2369] T 9c7a9e6a22314a258baa48a0ed5bf069 P a16f2984931649a2a594bdd90b4debae: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 03:25:55.370890 31695 raft_consensus.cc:482] T 9c7a9e6a22314a258baa48a0ed5bf069 P a16f2984931649a2a594bdd90b4debae [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 03:25:55.371222 31695 raft_consensus.cc:504] T 9c7a9e6a22314a258baa48a0ed5bf069 P a16f2984931649a2a594bdd90b4debae [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a16f2984931649a2a594bdd90b4debae" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 42247 } } peers { permanent_uuid: "2831154de99842959a694f57c980b448" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 33653 } } peers { permanent_uuid: "d9f56db87b6542bfaa31d2a10412e4be" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 33631 } }
I20260810 03:25:55.373019 31695 leader_election.cc:288] T 9c7a9e6a22314a258baa48a0ed5bf069 P a16f2984931649a2a594bdd90b4debae [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2831154de99842959a694f57c980b448 (127.27.161.1:33653), d9f56db87b6542bfaa31d2a10412e4be (127.27.161.2:33631)
I20260810 03:25:55.378867 31680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:55.373413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.161.3:50451 (local address 127.27.161.1:33653)
0810 03:25:55.373614 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0810 03:25:55.373631 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0810 03:25:55.373654 (+ 23us) server_negotiation.cc:407] Connection header received
0810 03:25:55.373922 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:25:55.373928 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:25:55.374034 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:25:55.374138 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:25:55.374526 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:55.375480 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:55.377230 (+ 1750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:55.378039 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:55.378190 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:25:55.378269 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0810 03:25:55.378508 (+ 239us) server_negotiation.cc:299] Negotiation successful
0810 03:25:55.378635 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260810 03:25:55.378909 31492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:55.373322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.1:33653 (local address 127.27.161.3:50451)
0810 03:25:55.373562 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0810 03:25:55.373585 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0810 03:25:55.373725 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:25:55.374165 (+ 440us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:25:55.374169 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:25:55.374197 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:25:55.374381 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:25:55.374389 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:25:55.375599 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:25:55.375604 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:25:55.377077 (+ 1473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:25:55.377090 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:25:55.378190 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:25:55.378196 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:25:55.378365 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:25:55.378390 (+ 25us) client_negotiation.cc:769] Sending connection context
0810 03:25:55.378471 (+ 81us) client_negotiation.cc:240] Negotiation successful
0810 03:25:55.378601 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":69}
I20260810 03:25:55.379578 31683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:55.373777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.161.3:44563 (local address 127.27.161.2:33631)
0810 03:25:55.374025 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0810 03:25:55.374030 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:25:55.374367 (+ 337us) server_negotiation.cc:407] Connection header received
0810 03:25:55.374575 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:25:55.374579 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:25:55.374642 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:25:55.374708 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:25:55.375282 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:55.376056 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:55.377847 (+ 1791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:55.378651 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:55.378799 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:25:55.378869 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0810 03:25:55.379235 (+ 366us) server_negotiation.cc:299] Negotiation successful
0810 03:25:55.379368 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260810 03:25:55.379681 31698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:55.373467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.2:33631 (local address 127.27.161.3:44563)
0810 03:25:55.374262 (+ 795us) negotiation.cc:107] Waiting for socket to connect
0810 03:25:55.374278 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 03:25:55.374408 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:25:55.374748 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:25:55.374752 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:25:55.374779 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:25:55.375137 (+ 358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:25:55.375149 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:25:55.376213 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:25:55.376217 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:25:55.377696 (+ 1479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:25:55.377708 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:25:55.378863 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:25:55.378869 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:25:55.379047 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:25:55.379080 (+ 33us) client_negotiation.cc:769] Sending connection context
0810 03:25:55.379198 (+ 118us) client_negotiation.cc:240] Negotiation successful
0810 03:25:55.379316 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":673,"mutex_wait_us":6,"thread_start_us":441,"threads_started":1}
I20260810 03:25:55.379894 31151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c7a9e6a22314a258baa48a0ed5bf069" candidate_uuid: "a16f2984931649a2a594bdd90b4debae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2831154de99842959a694f57c980b448" is_pre_election: true
I20260810 03:25:55.380451 31151 raft_consensus.cc:2454] T 9c7a9e6a22314a258baa48a0ed5bf069 P 2831154de99842959a694f57c980b448 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a16f2984931649a2a594bdd90b4debae in term 0.
I20260810 03:25:55.380599 31359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c7a9e6a22314a258baa48a0ed5bf069" candidate_uuid: "a16f2984931649a2a594bdd90b4debae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9f56db87b6542bfaa31d2a10412e4be" is_pre_election: true
I20260810 03:25:55.381058 31480 leader_election.cc:302] T 9c7a9e6a22314a258baa48a0ed5bf069 P a16f2984931649a2a594bdd90b4debae [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2831154de99842959a694f57c980b448, a16f2984931649a2a594bdd90b4debae; no voters:
I20260810 03:25:55.381068 31359 raft_consensus.cc:2454] T 9c7a9e6a22314a258baa48a0ed5bf069 P d9f56db87b6542bfaa31d2a10412e4be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a16f2984931649a2a594bdd90b4debae in term 0.
I20260810 03:25:55.381393 31695 raft_consensus.cc:2781] T 9c7a9e6a22314a258baa48a0ed5bf069 P a16f2984931649a2a594bdd90b4debae [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 03:25:55.381556 31695 raft_consensus.cc:482] T 9c7a9e6a22314a258baa48a0ed5bf069 P a16f2984931649a2a594bdd90b4debae [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 03:25:55.381670 31695 raft_consensus.cc:3037] T 9c7a9e6a22314a258baa48a0ed5bf069 P a16f2984931649a2a594bdd90b4debae [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:25:55.383908 31695 raft_consensus.cc:504] T 9c7a9e6a22314a258baa48a0ed5bf069 P a16f2984931649a2a594bdd90b4debae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a16f2984931649a2a594bdd90b4debae" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 42247 } } peers { permanent_uuid: "2831154de99842959a694f57c980b448" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 33653 } } peers { permanent_uuid: "d9f56db87b6542bfaa31d2a10412e4be" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 33631 } }
I20260810 03:25:55.384610 31695 leader_election.cc:288] T 9c7a9e6a22314a258baa48a0ed5bf069 P a16f2984931649a2a594bdd90b4debae [CANDIDATE]: Term 1 election: Requested vote from peers 2831154de99842959a694f57c980b448 (127.27.161.1:33653), d9f56db87b6542bfaa31d2a10412e4be (127.27.161.2:33631)
I20260810 03:25:55.384991 31151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c7a9e6a22314a258baa48a0ed5bf069" candidate_uuid: "a16f2984931649a2a594bdd90b4debae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2831154de99842959a694f57c980b448"
I20260810 03:25:55.385085 31359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c7a9e6a22314a258baa48a0ed5bf069" candidate_uuid: "a16f2984931649a2a594bdd90b4debae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9f56db87b6542bfaa31d2a10412e4be"
I20260810 03:25:55.385185 31151 raft_consensus.cc:3037] T 9c7a9e6a22314a258baa48a0ed5bf069 P 2831154de99842959a694f57c980b448 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:25:55.385300 31359 raft_consensus.cc:3037] T 9c7a9e6a22314a258baa48a0ed5bf069 P d9f56db87b6542bfaa31d2a10412e4be [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:25:55.387635 31151 raft_consensus.cc:2454] T 9c7a9e6a22314a258baa48a0ed5bf069 P 2831154de99842959a694f57c980b448 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a16f2984931649a2a594bdd90b4debae in term 1.
I20260810 03:25:55.388053 31480 leader_election.cc:302] T 9c7a9e6a22314a258baa48a0ed5bf069 P a16f2984931649a2a594bdd90b4debae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2831154de99842959a694f57c980b448, a16f2984931649a2a594bdd90b4debae; no voters:
I20260810 03:25:55.388161 31359 raft_consensus.cc:2454] T 9c7a9e6a22314a258baa48a0ed5bf069 P d9f56db87b6542bfaa31d2a10412e4be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a16f2984931649a2a594bdd90b4debae in term 1.
I20260810 03:25:55.388391 31695 raft_consensus.cc:2781] T 9c7a9e6a22314a258baa48a0ed5bf069 P a16f2984931649a2a594bdd90b4debae [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:25:55.388907 31695 raft_consensus.cc:686] T 9c7a9e6a22314a258baa48a0ed5bf069 P a16f2984931649a2a594bdd90b4debae [term 1 LEADER]: Becoming Leader. State: Replica: a16f2984931649a2a594bdd90b4debae, State: Running, Role: LEADER
I20260810 03:25:55.389389 31695 consensus_queue.cc:237] T 9c7a9e6a22314a258baa48a0ed5bf069 P a16f2984931649a2a594bdd90b4debae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a16f2984931649a2a594bdd90b4debae" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 42247 } } peers { permanent_uuid: "2831154de99842959a694f57c980b448" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 33653 } } peers { permanent_uuid: "d9f56db87b6542bfaa31d2a10412e4be" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 33631 } }
W20260810 03:25:55.394203 31227 tablet.cc:2369] T 9c7a9e6a22314a258baa48a0ed5bf069 P 2831154de99842959a694f57c980b448: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 03:25:55.394145 30906 catalog_manager.cc:5495] T 9c7a9e6a22314a258baa48a0ed5bf069 P a16f2984931649a2a594bdd90b4debae reported cstate change: term changed from 0 to 1, leader changed from <none> to a16f2984931649a2a594bdd90b4debae (127.27.161.3). New cstate: current_term: 1 leader_uuid: "a16f2984931649a2a594bdd90b4debae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a16f2984931649a2a594bdd90b4debae" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 42247 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2831154de99842959a694f57c980b448" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 33653 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d9f56db87b6542bfaa31d2a10412e4be" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 33631 } health_report { overall_health: UNKNOWN } } }
Aug 10 03:25:55 dist-test-slave-vpvm krb5kdc[30840](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332353, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.161.62@KRBTEST.COM for kudu/127.27.161.62@KRBTEST.COM
I20260810 03:25:55.458487 31710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:55.444865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:38085 (local address 127.0.0.1:55664)
0810 03:25:55.445547 (+ 682us) negotiation.cc:107] Waiting for socket to connect
0810 03:25:55.445566 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 03:25:55.445698 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:25:55.446033 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:25:55.446036 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:25:55.446702 (+ 666us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:25:55.446965 (+ 263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:25:55.446975 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:25:55.447963 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:25:55.447967 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:25:55.449031 (+ 1064us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:25:55.449044 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:25:55.449832 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:25:55.449837 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:25:55.449997 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:25:55.450635 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:25:55.450653 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:25:55.452971 (+ 2318us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:25:55.455810 (+ 2839us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:25:55.455824 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:25:55.455831 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:25:55.456368 (+ 537us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:25:55.456817 (+ 449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:25:55.456821 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:25:55.456824 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:25:55.456976 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:25:55.457508 (+ 532us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:25:55.457515 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:25:55.457678 (+ 163us) client_negotiation.cc:769] Sending connection context
0810 03:25:55.457820 (+ 142us) client_negotiation.cc:240] Negotiation successful
0810 03:25:55.458104 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":550,"thread_start_us":370,"threads_started":1}
I20260810 03:25:55.458525 31263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:55.444760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55664 (local address 127.27.161.62:38085)
0810 03:25:55.445018 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0810 03:25:55.445023 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:25:55.445641 (+ 618us) server_negotiation.cc:407] Connection header received
0810 03:25:55.445790 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:25:55.445793 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:25:55.445870 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:25:55.445979 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:25:55.447086 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:55.447844 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:55.449196 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:55.449632 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:55.449735 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:25:55.453181 (+ 3446us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:25:55.453186 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:25:55.453190 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:25:55.453221 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:25:55.455653 (+ 2432us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:25:55.456509 (+ 856us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:25:55.456514 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:25:55.456516 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:25:55.456685 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:25:55.457086 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:25:55.457090 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:25:55.457092 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:25:55.457391 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:25:55.457525 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0810 03:25:55.457917 (+ 392us) server_negotiation.cc:299] Negotiation successful
0810 03:25:55.458237 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":69,"server-negotiator.queue_time_us":83}
I20260810 03:25:55.464105 30906 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.27.161.62@KRBTEST.COM'} at 127.0.0.1:55664:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260810 03:25:55.465265 30906 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 03:25:55.472769 31126 tablet_service.cc:1467] Processing CreateTablet for tablet 31e3549f08314704a274b30b95b38f9e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=fb4812e13556429981cde56d7d39d0f4]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 03:25:55.473078 31331 tablet_service.cc:1467] Processing CreateTablet for tablet 31e3549f08314704a274b30b95b38f9e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=fb4812e13556429981cde56d7d39d0f4]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 03:25:55.473330 31559 tablet_service.cc:1467] Processing CreateTablet for tablet 31e3549f08314704a274b30b95b38f9e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=fb4812e13556429981cde56d7d39d0f4]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 03:25:55.473914 31331 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31e3549f08314704a274b30b95b38f9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:25:55.473932 31126 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31e3549f08314704a274b30b95b38f9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:25:55.474170 31559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31e3549f08314704a274b30b95b38f9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:25:55.481865 31688 tablet_bootstrap.cc:492] T 31e3549f08314704a274b30b95b38f9e P 2831154de99842959a694f57c980b448: Bootstrap starting.
I20260810 03:25:55.481941 31687 tablet_bootstrap.cc:492] T 31e3549f08314704a274b30b95b38f9e P a16f2984931649a2a594bdd90b4debae: Bootstrap starting.
I20260810 03:25:55.483439 31688 tablet_bootstrap.cc:654] T 31e3549f08314704a274b30b95b38f9e P 2831154de99842959a694f57c980b448: Neither blocks nor log segments found. Creating new log.
I20260810 03:25:55.483851 31687 tablet_bootstrap.cc:654] T 31e3549f08314704a274b30b95b38f9e P a16f2984931649a2a594bdd90b4debae: Neither blocks nor log segments found. Creating new log.
I20260810 03:25:55.485538 31684 tablet_bootstrap.cc:492] T 31e3549f08314704a274b30b95b38f9e P d9f56db87b6542bfaa31d2a10412e4be: Bootstrap starting.
I20260810 03:25:55.485899 31688 tablet_bootstrap.cc:492] T 31e3549f08314704a274b30b95b38f9e P 2831154de99842959a694f57c980b448: No bootstrap required, opened a new log
I20260810 03:25:55.486089 31688 ts_tablet_manager.cc:1397] T 31e3549f08314704a274b30b95b38f9e P 2831154de99842959a694f57c980b448: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.004s
I20260810 03:25:55.486867 31688 raft_consensus.cc:348] T 31e3549f08314704a274b30b95b38f9e P 2831154de99842959a694f57c980b448 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2831154de99842959a694f57c980b448" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 33653 } } peers { permanent_uuid: "d9f56db87b6542bfaa31d2a10412e4be" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 33631 } } peers { permanent_uuid: "a16f2984931649a2a594bdd90b4debae" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 42247 } }
I20260810 03:25:55.487080 31688 raft_consensus.cc:374] T 31e3549f08314704a274b30b95b38f9e P 2831154de99842959a694f57c980b448 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:25:55.487174 31688 raft_consensus.cc:729] T 31e3549f08314704a274b30b95b38f9e P 2831154de99842959a694f57c980b448 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2831154de99842959a694f57c980b448, State: Initialized, Role: FOLLOWER
I20260810 03:25:55.487385 31684 tablet_bootstrap.cc:654] T 31e3549f08314704a274b30b95b38f9e P d9f56db87b6542bfaa31d2a10412e4be: Neither blocks nor log segments found. Creating new log.
I20260810 03:25:55.487462 31688 consensus_queue.cc:260] T 31e3549f08314704a274b30b95b38f9e P 2831154de99842959a694f57c980b448 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2831154de99842959a694f57c980b448" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 33653 } } peers { permanent_uuid: "d9f56db87b6542bfaa31d2a10412e4be" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 33631 } } peers { permanent_uuid: "a16f2984931649a2a594bdd90b4debae" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 42247 } }
I20260810 03:25:55.487962 31688 ts_tablet_manager.cc:1428] T 31e3549f08314704a274b30b95b38f9e P 2831154de99842959a694f57c980b448: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20260810 03:25:55.488569 31696 tablet_replica.cc:439] T 31e3549f08314704a274b30b95b38f9e P 2831154de99842959a694f57c980b448: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2831154de99842959a694f57c980b448" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 33653 } } peers { permanent_uuid: "d9f56db87b6542bfaa31d2a10412e4be" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 33631 } } peers { permanent_uuid: "a16f2984931649a2a594bdd90b4debae" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 42247 } } }
I20260810 03:25:55.488970 31696 tablet_replica.cc:442] T 31e3549f08314704a274b30b95b38f9e P 2831154de99842959a694f57c980b448: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 03:25:55.490556 31684 tablet_bootstrap.cc:492] T 31e3549f08314704a274b30b95b38f9e P d9f56db87b6542bfaa31d2a10412e4be: No bootstrap required, opened a new log
I20260810 03:25:55.490708 31684 ts_tablet_manager.cc:1397] T 31e3549f08314704a274b30b95b38f9e P d9f56db87b6542bfaa31d2a10412e4be: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260810 03:25:55.491485 31684 raft_consensus.cc:348] T 31e3549f08314704a274b30b95b38f9e P d9f56db87b6542bfaa31d2a10412e4be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2831154de99842959a694f57c980b448" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 33653 } } peers { permanent_uuid: "d9f56db87b6542bfaa31d2a10412e4be" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 33631 } } peers { permanent_uuid: "a16f2984931649a2a594bdd90b4debae" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 42247 } }
I20260810 03:25:55.491748 31684 raft_consensus.cc:374] T 31e3549f08314704a274b30b95b38f9e P d9f56db87b6542bfaa31d2a10412e4be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:25:55.491837 31684 raft_consensus.cc:729] T 31e3549f08314704a274b30b95b38f9e P d9f56db87b6542bfaa31d2a10412e4be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9f56db87b6542bfaa31d2a10412e4be, State: Initialized, Role: FOLLOWER
I20260810 03:25:55.492077 31684 consensus_queue.cc:260] T 31e3549f08314704a274b30b95b38f9e P d9f56db87b6542bfaa31d2a10412e4be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2831154de99842959a694f57c980b448" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 33653 } } peers { permanent_uuid: "d9f56db87b6542bfaa31d2a10412e4be" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 33631 } } peers { permanent_uuid: "a16f2984931649a2a594bdd90b4debae" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 42247 } }
I20260810 03:25:55.492527 31693 tablet_replica.cc:439] T 31e3549f08314704a274b30b95b38f9e P d9f56db87b6542bfaa31d2a10412e4be: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2831154de99842959a694f57c980b448" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 33653 } } peers { permanent_uuid: "d9f56db87b6542bfaa31d2a10412e4be" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 33631 } } peers { permanent_uuid: "a16f2984931649a2a594bdd90b4debae" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 42247 } } }
I20260810 03:25:55.492913 31693 tablet_replica.cc:442] T 31e3549f08314704a274b30b95b38f9e P d9f56db87b6542bfaa31d2a10412e4be: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 03:25:55.494083 31684 ts_tablet_manager.cc:1428] T 31e3549f08314704a274b30b95b38f9e P d9f56db87b6542bfaa31d2a10412e4be: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260810 03:25:55.495018 31687 tablet_bootstrap.cc:492] T 31e3549f08314704a274b30b95b38f9e P a16f2984931649a2a594bdd90b4debae: No bootstrap required, opened a new log
I20260810 03:25:55.495201 31687 ts_tablet_manager.cc:1397] T 31e3549f08314704a274b30b95b38f9e P a16f2984931649a2a594bdd90b4debae: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.000s
I20260810 03:25:55.496184 31687 raft_consensus.cc:348] T 31e3549f08314704a274b30b95b38f9e P a16f2984931649a2a594bdd90b4debae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2831154de99842959a694f57c980b448" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 33653 } } peers { permanent_uuid: "d9f56db87b6542bfaa31d2a10412e4be" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 33631 } } peers { permanent_uuid: "a16f2984931649a2a594bdd90b4debae" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 42247 } }
I20260810 03:25:55.496397 31687 raft_consensus.cc:374] T 31e3549f08314704a274b30b95b38f9e P a16f2984931649a2a594bdd90b4debae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:25:55.496501 31687 raft_consensus.cc:729] T 31e3549f08314704a274b30b95b38f9e P a16f2984931649a2a594bdd90b4debae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a16f2984931649a2a594bdd90b4debae, State: Initialized, Role: FOLLOWER
I20260810 03:25:55.496747 31687 consensus_queue.cc:260] T 31e3549f08314704a274b30b95b38f9e P a16f2984931649a2a594bdd90b4debae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2831154de99842959a694f57c980b448" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 33653 } } peers { permanent_uuid: "d9f56db87b6542bfaa31d2a10412e4be" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 33631 } } peers { permanent_uuid: "a16f2984931649a2a594bdd90b4debae" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 42247 } }
I20260810 03:25:55.497234 31695 tablet_replica.cc:439] T 31e3549f08314704a274b30b95b38f9e P a16f2984931649a2a594bdd90b4debae: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2831154de99842959a694f57c980b448" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 33653 } } peers { permanent_uuid: "d9f56db87b6542bfaa31d2a10412e4be" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 33631 } } peers { permanent_uuid: "a16f2984931649a2a594bdd90b4debae" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 42247 } } }
I20260810 03:25:55.497426 31687 ts_tablet_manager.cc:1428] T 31e3549f08314704a274b30b95b38f9e P a16f2984931649a2a594bdd90b4debae: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260810 03:25:55.497581 31695 tablet_replica.cc:442] T 31e3549f08314704a274b30b95b38f9e P a16f2984931649a2a594bdd90b4debae: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 03:25:55.594339 31695 raft_consensus.cc:482] T 31e3549f08314704a274b30b95b38f9e P a16f2984931649a2a594bdd90b4debae [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 03:25:55.594576 31695 raft_consensus.cc:504] T 31e3549f08314704a274b30b95b38f9e P a16f2984931649a2a594bdd90b4debae [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2831154de99842959a694f57c980b448" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 33653 } } peers { permanent_uuid: "d9f56db87b6542bfaa31d2a10412e4be" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 33631 } } peers { permanent_uuid: "a16f2984931649a2a594bdd90b4debae" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 42247 } }
I20260810 03:25:55.595218 31695 leader_election.cc:288] T 31e3549f08314704a274b30b95b38f9e P a16f2984931649a2a594bdd90b4debae [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2831154de99842959a694f57c980b448 (127.27.161.1:33653), d9f56db87b6542bfaa31d2a10412e4be (127.27.161.2:33631)
I20260810 03:25:55.595867 31151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31e3549f08314704a274b30b95b38f9e" candidate_uuid: "a16f2984931649a2a594bdd90b4debae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2831154de99842959a694f57c980b448" is_pre_election: true
I20260810 03:25:55.595870 31359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31e3549f08314704a274b30b95b38f9e" candidate_uuid: "a16f2984931649a2a594bdd90b4debae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9f56db87b6542bfaa31d2a10412e4be" is_pre_election: true
I20260810 03:25:55.596119 31151 raft_consensus.cc:2454] T 31e3549f08314704a274b30b95b38f9e P 2831154de99842959a694f57c980b448 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a16f2984931649a2a594bdd90b4debae in term 0.
I20260810 03:25:55.596125 31359 raft_consensus.cc:2454] T 31e3549f08314704a274b30b95b38f9e P d9f56db87b6542bfaa31d2a10412e4be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a16f2984931649a2a594bdd90b4debae in term 0.
I20260810 03:25:55.596565 31480 leader_election.cc:302] T 31e3549f08314704a274b30b95b38f9e P a16f2984931649a2a594bdd90b4debae [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2831154de99842959a694f57c980b448, a16f2984931649a2a594bdd90b4debae; no voters:
I20260810 03:25:55.596858 31695 raft_consensus.cc:2781] T 31e3549f08314704a274b30b95b38f9e P a16f2984931649a2a594bdd90b4debae [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 03:25:55.596946 31695 raft_consensus.cc:482] T 31e3549f08314704a274b30b95b38f9e P a16f2984931649a2a594bdd90b4debae [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 03:25:55.597030 31695 raft_consensus.cc:3037] T 31e3549f08314704a274b30b95b38f9e P a16f2984931649a2a594bdd90b4debae [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:25:55.599052 31695 raft_consensus.cc:504] T 31e3549f08314704a274b30b95b38f9e P a16f2984931649a2a594bdd90b4debae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2831154de99842959a694f57c980b448" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 33653 } } peers { permanent_uuid: "d9f56db87b6542bfaa31d2a10412e4be" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 33631 } } peers { permanent_uuid: "a16f2984931649a2a594bdd90b4debae" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 42247 } }
I20260810 03:25:55.599614 31695 leader_election.cc:288] T 31e3549f08314704a274b30b95b38f9e P a16f2984931649a2a594bdd90b4debae [CANDIDATE]: Term 1 election: Requested vote from peers 2831154de99842959a694f57c980b448 (127.27.161.1:33653), d9f56db87b6542bfaa31d2a10412e4be (127.27.161.2:33631)
I20260810 03:25:55.600195 31359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31e3549f08314704a274b30b95b38f9e" candidate_uuid: "a16f2984931649a2a594bdd90b4debae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9f56db87b6542bfaa31d2a10412e4be"
I20260810 03:25:55.600219 31151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31e3549f08314704a274b30b95b38f9e" candidate_uuid: "a16f2984931649a2a594bdd90b4debae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2831154de99842959a694f57c980b448"
I20260810 03:25:55.600418 31359 raft_consensus.cc:3037] T 31e3549f08314704a274b30b95b38f9e P d9f56db87b6542bfaa31d2a10412e4be [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:25:55.600423 31151 raft_consensus.cc:3037] T 31e3549f08314704a274b30b95b38f9e P 2831154de99842959a694f57c980b448 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:25:55.602581 31151 raft_consensus.cc:2454] T 31e3549f08314704a274b30b95b38f9e P 2831154de99842959a694f57c980b448 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a16f2984931649a2a594bdd90b4debae in term 1.
I20260810 03:25:55.602945 31359 raft_consensus.cc:2454] T 31e3549f08314704a274b30b95b38f9e P d9f56db87b6542bfaa31d2a10412e4be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a16f2984931649a2a594bdd90b4debae in term 1.
I20260810 03:25:55.603065 31480 leader_election.cc:302] T 31e3549f08314704a274b30b95b38f9e P a16f2984931649a2a594bdd90b4debae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2831154de99842959a694f57c980b448, a16f2984931649a2a594bdd90b4debae; no voters:
I20260810 03:25:55.603430 31695 raft_consensus.cc:2781] T 31e3549f08314704a274b30b95b38f9e P a16f2984931649a2a594bdd90b4debae [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:25:55.603631 31695 raft_consensus.cc:686] T 31e3549f08314704a274b30b95b38f9e P a16f2984931649a2a594bdd90b4debae [term 1 LEADER]: Becoming Leader. State: Replica: a16f2984931649a2a594bdd90b4debae, State: Running, Role: LEADER
I20260810 03:25:55.603900 31695 consensus_queue.cc:237] T 31e3549f08314704a274b30b95b38f9e P a16f2984931649a2a594bdd90b4debae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2831154de99842959a694f57c980b448" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 33653 } } peers { permanent_uuid: "d9f56db87b6542bfaa31d2a10412e4be" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 33631 } } peers { permanent_uuid: "a16f2984931649a2a594bdd90b4debae" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 42247 } }
I20260810 03:25:55.605477 31700 tablet_replica.cc:439] T 31e3549f08314704a274b30b95b38f9e P a16f2984931649a2a594bdd90b4debae: TxnStatusTablet state changed. Reason: New leader a16f2984931649a2a594bdd90b4debae. Latest consensus state: current_term: 1 leader_uuid: "a16f2984931649a2a594bdd90b4debae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2831154de99842959a694f57c980b448" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 33653 } } peers { permanent_uuid: "d9f56db87b6542bfaa31d2a10412e4be" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 33631 } } peers { permanent_uuid: "a16f2984931649a2a594bdd90b4debae" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 42247 } } }
I20260810 03:25:55.605772 31700 tablet_replica.cc:442] T 31e3549f08314704a274b30b95b38f9e P a16f2984931649a2a594bdd90b4debae: This TxnStatusTablet replica's current role is: LEADER
I20260810 03:25:55.606451 31717 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260810 03:25:55.606756 31717 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260810 03:25:55.607367 30908 catalog_manager.cc:5495] T 31e3549f08314704a274b30b95b38f9e P a16f2984931649a2a594bdd90b4debae reported cstate change: term changed from 0 to 1, leader changed from <none> to a16f2984931649a2a594bdd90b4debae (127.27.161.3). New cstate: current_term: 1 leader_uuid: "a16f2984931649a2a594bdd90b4debae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2831154de99842959a694f57c980b448" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 33653 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d9f56db87b6542bfaa31d2a10412e4be" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 33631 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a16f2984931649a2a594bdd90b4debae" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 42247 } health_report { overall_health: HEALTHY } } }
I20260810 03:25:55.653286 31710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:55.646610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.3:42247 (local address 127.0.0.1:44202)
0810 03:25:55.646886 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0810 03:25:55.646911 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0810 03:25:55.647066 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:25:55.647682 (+ 616us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:25:55.647685 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:25:55.647696 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 03:25:55.647866 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:25:55.647873 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:25:55.649325 (+ 1452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:25:55.649328 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:25:55.650384 (+ 1056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:25:55.650399 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:25:55.651392 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:25:55.651396 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:25:55.651543 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:25:55.651609 (+ 66us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 03:25:55.652804 (+ 1195us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 03:25:55.652827 (+ 23us) client_negotiation.cc:769] Sending connection context
0810 03:25:55.652905 (+ 78us) client_negotiation.cc:240] Negotiation successful
0810 03:25:55.653043 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20260810 03:25:55.653364 31721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:55.646683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44202 (local address 127.27.161.3:42247)
0810 03:25:55.647221 (+ 538us) server_negotiation.cc:206] Beginning negotiation
0810 03:25:55.647228 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:25:55.647248 (+ 20us) server_negotiation.cc:407] Connection header received
0810 03:25:55.647351 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:25:55.647357 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:25:55.647469 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:25:55.647674 (+ 205us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 03:25:55.648012 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:55.649202 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:55.650526 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:55.651242 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:55.651373 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:25:55.651824 (+ 451us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 03:25:55.652688 (+ 864us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 03:25:55.652854 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0810 03:25:55.652970 (+ 116us) server_negotiation.cc:299] Negotiation successful
0810 03:25:55.653068 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":395,"thread_start_us":260,"threads_started":1}
I20260810 03:25:55.662487 31359 raft_consensus.cc:1261] T 31e3549f08314704a274b30b95b38f9e P d9f56db87b6542bfaa31d2a10412e4be [term 1 FOLLOWER]: Refusing update from remote peer a16f2984931649a2a594bdd90b4debae: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:25:55.662604 31151 raft_consensus.cc:1261] T 31e3549f08314704a274b30b95b38f9e P 2831154de99842959a694f57c980b448 [term 1 FOLLOWER]: Refusing update from remote peer a16f2984931649a2a594bdd90b4debae: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:25:55.663486 31700 consensus_queue.cc:1035] T 31e3549f08314704a274b30b95b38f9e P a16f2984931649a2a594bdd90b4debae [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9f56db87b6542bfaa31d2a10412e4be" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 33631 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:25:55.663838 31695 consensus_queue.cc:1035] T 31e3549f08314704a274b30b95b38f9e P a16f2984931649a2a594bdd90b4debae [LEADER]: Connected to new peer: Peer: permanent_uuid: "2831154de99842959a694f57c980b448" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 33653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:25:55.676957 31693 tablet_replica.cc:439] T 31e3549f08314704a274b30b95b38f9e P d9f56db87b6542bfaa31d2a10412e4be: TxnStatusTablet state changed. Reason: New leader a16f2984931649a2a594bdd90b4debae. Latest consensus state: current_term: 1 leader_uuid: "a16f2984931649a2a594bdd90b4debae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2831154de99842959a694f57c980b448" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 33653 } } peers { permanent_uuid: "d9f56db87b6542bfaa31d2a10412e4be" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 33631 } } peers { permanent_uuid: "a16f2984931649a2a594bdd90b4debae" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 42247 } } }
I20260810 03:25:55.677310 31693 tablet_replica.cc:442] T 31e3549f08314704a274b30b95b38f9e P d9f56db87b6542bfaa31d2a10412e4be: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 03:25:55.679593 31696 tablet_replica.cc:439] T 31e3549f08314704a274b30b95b38f9e P 2831154de99842959a694f57c980b448: TxnStatusTablet state changed. Reason: New leader a16f2984931649a2a594bdd90b4debae. Latest consensus state: current_term: 1 leader_uuid: "a16f2984931649a2a594bdd90b4debae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2831154de99842959a694f57c980b448" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 33653 } } peers { permanent_uuid: "d9f56db87b6542bfaa31d2a10412e4be" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 33631 } } peers { permanent_uuid: "a16f2984931649a2a594bdd90b4debae" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 42247 } } }
I20260810 03:25:55.679903 31696 tablet_replica.cc:442] T 31e3549f08314704a274b30b95b38f9e P 2831154de99842959a694f57c980b448: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 03:25:55.680358 31722 mvcc.cc:205] Tried to move back new op lower bound from 7316817328782958592 to 7316817328556109824. Current Snapshot: MvccSnapshot[applied={T|T < 7316817328782958592}]
I20260810 03:25:55.687516 31723 mvcc.cc:205] Tried to move back new op lower bound from 7316817328782958592 to 7316817328556109824. Current Snapshot: MvccSnapshot[applied={T|T < 7316817328782958592}]
I20260810 03:25:55.688330 31696 tablet_replica.cc:439] T 31e3549f08314704a274b30b95b38f9e P 2831154de99842959a694f57c980b448: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a16f2984931649a2a594bdd90b4debae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2831154de99842959a694f57c980b448" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 33653 } } peers { permanent_uuid: "d9f56db87b6542bfaa31d2a10412e4be" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 33631 } } peers { permanent_uuid: "a16f2984931649a2a594bdd90b4debae" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 42247 } } }
I20260810 03:25:55.688597 31696 tablet_replica.cc:442] T 31e3549f08314704a274b30b95b38f9e P 2831154de99842959a694f57c980b448: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 03:25:55.689669 31724 mvcc.cc:205] Tried to move back new op lower bound from 7316817328782958592 to 7316817328556109824. Current Snapshot: MvccSnapshot[applied={T|T < 7316817328782958592}]
I20260810 03:25:55.691175 31695 tablet_replica.cc:439] T 31e3549f08314704a274b30b95b38f9e P a16f2984931649a2a594bdd90b4debae: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a16f2984931649a2a594bdd90b4debae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2831154de99842959a694f57c980b448" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 33653 } } peers { permanent_uuid: "d9f56db87b6542bfaa31d2a10412e4be" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 33631 } } peers { permanent_uuid: "a16f2984931649a2a594bdd90b4debae" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 42247 } } }
I20260810 03:25:55.691509 31695 tablet_replica.cc:442] T 31e3549f08314704a274b30b95b38f9e P a16f2984931649a2a594bdd90b4debae: This TxnStatusTablet replica's current role is: LEADER
I20260810 03:25:55.691896 31695 tablet_replica.cc:439] T 31e3549f08314704a274b30b95b38f9e P a16f2984931649a2a594bdd90b4debae: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a16f2984931649a2a594bdd90b4debae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2831154de99842959a694f57c980b448" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 33653 } } peers { permanent_uuid: "d9f56db87b6542bfaa31d2a10412e4be" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 33631 } } peers { permanent_uuid: "a16f2984931649a2a594bdd90b4debae" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 42247 } } }
I20260810 03:25:55.692188 31695 tablet_replica.cc:442] T 31e3549f08314704a274b30b95b38f9e P a16f2984931649a2a594bdd90b4debae: This TxnStatusTablet replica's current role is: LEADER
I20260810 03:25:55.698175 31693 tablet_replica.cc:439] T 31e3549f08314704a274b30b95b38f9e P d9f56db87b6542bfaa31d2a10412e4be: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a16f2984931649a2a594bdd90b4debae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2831154de99842959a694f57c980b448" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 33653 } } peers { permanent_uuid: "d9f56db87b6542bfaa31d2a10412e4be" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 33631 } } peers { permanent_uuid: "a16f2984931649a2a594bdd90b4debae" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 42247 } } }
I20260810 03:25:55.698458 31693 tablet_replica.cc:442] T 31e3549f08314704a274b30b95b38f9e P d9f56db87b6542bfaa31d2a10412e4be: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 03:25:55.708046 31721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:55.701879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44214 (local address 127.27.161.3:42247)
0810 03:25:55.702110 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0810 03:25:55.702117 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:25:55.702447 (+ 330us) server_negotiation.cc:407] Connection header received
0810 03:25:55.702610 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:25:55.702616 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:25:55.702714 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:25:55.702826 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 03:25:55.703355 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:55.704517 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:55.705944 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:55.706629 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:55.706771 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:25:55.707159 (+ 388us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 03:25:55.707311 (+ 152us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 03:25:55.707410 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0810 03:25:55.707609 (+ 199us) server_negotiation.cc:299] Negotiation successful
0810 03:25:55.707704 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105}
I20260810 03:25:55.727945 31721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:55.721020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.161.3:45177 (local address 127.27.161.3:42247)
0810 03:25:55.721249 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0810 03:25:55.721256 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:25:55.721890 (+ 634us) server_negotiation.cc:407] Connection header received
0810 03:25:55.722114 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:25:55.722120 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:25:55.722217 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:25:55.722336 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 03:25:55.722810 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:55.723976 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:55.725611 (+ 1635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:55.726397 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:55.726534 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:25:55.727035 (+ 501us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 03:25:55.727213 (+ 178us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 03:25:55.727308 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0810 03:25:55.727488 (+ 180us) server_negotiation.cc:299] Negotiation successful
0810 03:25:55.727580 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260810 03:25:55.727967 31748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:55.721020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.3:42247 (local address 127.27.161.3:45177)
0810 03:25:55.721788 (+ 768us) negotiation.cc:107] Waiting for socket to connect
0810 03:25:55.721807 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 03:25:55.721944 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:25:55.722345 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:25:55.722349 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:25:55.722359 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 03:25:55.722671 (+ 312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:25:55.722679 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:25:55.724142 (+ 1463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:25:55.724148 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:25:55.725467 (+ 1319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:25:55.725481 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:25:55.726599 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:25:55.726612 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:25:55.726759 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:25:55.726829 (+ 70us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 03:25:55.727345 (+ 516us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 03:25:55.727358 (+ 13us) client_negotiation.cc:769] Sending connection context
0810 03:25:55.727447 (+ 89us) client_negotiation.cc:240] Negotiation successful
0810 03:25:55.727606 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":574,"mutex_wait_us":3,"thread_start_us":391,"threads_started":1}
I20260810 03:25:55.739886 31151 raft_consensus.cc:1261] T 9c7a9e6a22314a258baa48a0ed5bf069 P 2831154de99842959a694f57c980b448 [term 1 FOLLOWER]: Refusing update from remote peer a16f2984931649a2a594bdd90b4debae: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:25:55.739938 31359 raft_consensus.cc:1261] T 9c7a9e6a22314a258baa48a0ed5bf069 P d9f56db87b6542bfaa31d2a10412e4be [term 1 FOLLOWER]: Refusing update from remote peer a16f2984931649a2a594bdd90b4debae: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:25:55.740479 31700 consensus_queue.cc:1035] T 9c7a9e6a22314a258baa48a0ed5bf069 P a16f2984931649a2a594bdd90b4debae [LEADER]: Connected to new peer: Peer: permanent_uuid: "2831154de99842959a694f57c980b448" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 33653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:25:55.740859 31695 consensus_queue.cc:1035] T 9c7a9e6a22314a258baa48a0ed5bf069 P a16f2984931649a2a594bdd90b4debae [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9f56db87b6542bfaa31d2a10412e4be" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 33631 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:25:55.816710 30908 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55662:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 03:25:55.816959 30908 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55662:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 03:25:55.820823 30908 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0733cfa08dc743d8a587b321ff9eba88: Sending DeleteTablet for 3 replicas of tablet 9c7a9e6a22314a258baa48a0ed5bf069
I20260810 03:25:55.822062 31126 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c7a9e6a22314a258baa48a0ed5bf069 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:25:55 UTC) from {username='kudu', principal='kudu/127.27.161.62@KRBTEST.COM'} at 127.0.0.1:59472
I20260810 03:25:55.822132 31558 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c7a9e6a22314a258baa48a0ed5bf069 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:25:55 UTC) from {username='kudu', principal='kudu/127.27.161.62@KRBTEST.COM'} at 127.0.0.1:44192
I20260810 03:25:55.823527 31331 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c7a9e6a22314a258baa48a0ed5bf069 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:25:55 UTC) from {username='kudu', principal='kudu/127.27.161.62@KRBTEST.COM'} at 127.0.0.1:40112
I20260810 03:25:55.823426 30908 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55662:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 03:25:55.824221 30908 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 03:25:55.825970 31756 tablet_replica.cc:330] T 9c7a9e6a22314a258baa48a0ed5bf069 P d9f56db87b6542bfaa31d2a10412e4be: stopping tablet replica
I20260810 03:25:55.826673 31756 raft_consensus.cc:2229] T 9c7a9e6a22314a258baa48a0ed5bf069 P d9f56db87b6542bfaa31d2a10412e4be [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 03:25:55.827085 31756 raft_consensus.cc:2258] T 9c7a9e6a22314a258baa48a0ed5bf069 P d9f56db87b6542bfaa31d2a10412e4be [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:25:55.827883 31754 tablet_replica.cc:330] T 9c7a9e6a22314a258baa48a0ed5bf069 P 2831154de99842959a694f57c980b448: stopping tablet replica
I20260810 03:25:55.828320 31754 raft_consensus.cc:2229] T 9c7a9e6a22314a258baa48a0ed5bf069 P 2831154de99842959a694f57c980b448 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 03:25:55.828698 31754 raft_consensus.cc:2258] T 9c7a9e6a22314a258baa48a0ed5bf069 P 2831154de99842959a694f57c980b448 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:25:55.830905 31754 ts_tablet_manager.cc:1905] T 9c7a9e6a22314a258baa48a0ed5bf069 P 2831154de99842959a694f57c980b448: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:25:55.834427 31559 tablet_service.cc:1467] Processing CreateTablet for tablet 0bd9248b18c74e949e230c57b0ba90a8 (DEFAULT_TABLE table=test-table [id=0b7e97ce95f74a95864447818102993b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:25:55.834966 31559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0bd9248b18c74e949e230c57b0ba90a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:25:55.839048 31755 tablet_replica.cc:330] T 9c7a9e6a22314a258baa48a0ed5bf069 P a16f2984931649a2a594bdd90b4debae: stopping tablet replica
I20260810 03:25:55.839447 31754 ts_tablet_manager.cc:1918] T 9c7a9e6a22314a258baa48a0ed5bf069 P 2831154de99842959a694f57c980b448: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 03:25:55.839512 31755 raft_consensus.cc:2229] T 9c7a9e6a22314a258baa48a0ed5bf069 P a16f2984931649a2a594bdd90b4debae [term 1 LEADER]: Raft consensus shutting down.
I20260810 03:25:55.839653 31754 log.cc:1196] T 9c7a9e6a22314a258baa48a0ed5bf069 P 2831154de99842959a694f57c980b448: Deleting WAL directory at /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786332336319544-28292-0/minicluster-data/ts-0/wal/wals/9c7a9e6a22314a258baa48a0ed5bf069
I20260810 03:25:55.839658 31756 ts_tablet_manager.cc:1905] T 9c7a9e6a22314a258baa48a0ed5bf069 P d9f56db87b6542bfaa31d2a10412e4be: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:25:55.840189 31754 ts_tablet_manager.cc:1939] T 9c7a9e6a22314a258baa48a0ed5bf069 P 2831154de99842959a694f57c980b448: Deleting consensus metadata
I20260810 03:25:55.840250 31755 raft_consensus.cc:2258] T 9c7a9e6a22314a258baa48a0ed5bf069 P a16f2984931649a2a594bdd90b4debae [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:25:55.840699 31126 tablet_service.cc:1467] Processing CreateTablet for tablet 0bd9248b18c74e949e230c57b0ba90a8 (DEFAULT_TABLE table=test-table [id=0b7e97ce95f74a95864447818102993b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:25:55.841223 31126 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0bd9248b18c74e949e230c57b0ba90a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:25:55.841975 31331 tablet_service.cc:1467] Processing CreateTablet for tablet 0bd9248b18c74e949e230c57b0ba90a8 (DEFAULT_TABLE table=test-table [id=0b7e97ce95f74a95864447818102993b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:25:55.842051 30887 catalog_manager.cc:4841] TS 2831154de99842959a694f57c980b448 (127.27.161.1:33653): tablet 9c7a9e6a22314a258baa48a0ed5bf069 (table test-table [id=5cf9906ca78a4ac684e6520a2f83a069]) successfully deleted
I20260810 03:25:55.842478 31331 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0bd9248b18c74e949e230c57b0ba90a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:25:55.843174 31755 ts_tablet_manager.cc:1905] T 9c7a9e6a22314a258baa48a0ed5bf069 P a16f2984931649a2a594bdd90b4debae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:25:55.847793 31756 ts_tablet_manager.cc:1918] T 9c7a9e6a22314a258baa48a0ed5bf069 P d9f56db87b6542bfaa31d2a10412e4be: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 03:25:55.848012 31756 log.cc:1196] T 9c7a9e6a22314a258baa48a0ed5bf069 P d9f56db87b6542bfaa31d2a10412e4be: Deleting WAL directory at /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786332336319544-28292-0/minicluster-data/ts-1/wal/wals/9c7a9e6a22314a258baa48a0ed5bf069
I20260810 03:25:55.848585 31756 ts_tablet_manager.cc:1939] T 9c7a9e6a22314a258baa48a0ed5bf069 P d9f56db87b6542bfaa31d2a10412e4be: Deleting consensus metadata
I20260810 03:25:55.851068 31755 ts_tablet_manager.cc:1918] T 9c7a9e6a22314a258baa48a0ed5bf069 P a16f2984931649a2a594bdd90b4debae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 03:25:55.851266 31755 log.cc:1196] T 9c7a9e6a22314a258baa48a0ed5bf069 P a16f2984931649a2a594bdd90b4debae: Deleting WAL directory at /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786332336319544-28292-0/minicluster-data/ts-2/wal/wals/9c7a9e6a22314a258baa48a0ed5bf069
I20260810 03:25:55.851824 31755 ts_tablet_manager.cc:1939] T 9c7a9e6a22314a258baa48a0ed5bf069 P a16f2984931649a2a594bdd90b4debae: Deleting consensus metadata
I20260810 03:25:55.852566 31687 tablet_bootstrap.cc:492] T 0bd9248b18c74e949e230c57b0ba90a8 P a16f2984931649a2a594bdd90b4debae: Bootstrap starting.
I20260810 03:25:55.852888 31684 tablet_bootstrap.cc:492] T 0bd9248b18c74e949e230c57b0ba90a8 P d9f56db87b6542bfaa31d2a10412e4be: Bootstrap starting.
I20260810 03:25:55.853657 30888 catalog_manager.cc:4841] TS d9f56db87b6542bfaa31d2a10412e4be (127.27.161.2:33631): tablet 9c7a9e6a22314a258baa48a0ed5bf069 (table test-table [id=5cf9906ca78a4ac684e6520a2f83a069]) successfully deleted
I20260810 03:25:55.853917 30888 catalog_manager.cc:4841] TS a16f2984931649a2a594bdd90b4debae (127.27.161.3:42247): tablet 9c7a9e6a22314a258baa48a0ed5bf069 (table test-table [id=5cf9906ca78a4ac684e6520a2f83a069]) successfully deleted
I20260810 03:25:55.854336 31687 tablet_bootstrap.cc:654] T 0bd9248b18c74e949e230c57b0ba90a8 P a16f2984931649a2a594bdd90b4debae: Neither blocks nor log segments found. Creating new log.
I20260810 03:25:55.854614 31684 tablet_bootstrap.cc:654] T 0bd9248b18c74e949e230c57b0ba90a8 P d9f56db87b6542bfaa31d2a10412e4be: Neither blocks nor log segments found. Creating new log.
I20260810 03:25:55.855404 31688 tablet_bootstrap.cc:492] T 0bd9248b18c74e949e230c57b0ba90a8 P 2831154de99842959a694f57c980b448: Bootstrap starting.
I20260810 03:25:55.857215 31688 tablet_bootstrap.cc:654] T 0bd9248b18c74e949e230c57b0ba90a8 P 2831154de99842959a694f57c980b448: Neither blocks nor log segments found. Creating new log.
I20260810 03:25:55.866694 31684 tablet_bootstrap.cc:492] T 0bd9248b18c74e949e230c57b0ba90a8 P d9f56db87b6542bfaa31d2a10412e4be: No bootstrap required, opened a new log
I20260810 03:25:55.866945 31684 ts_tablet_manager.cc:1397] T 0bd9248b18c74e949e230c57b0ba90a8 P d9f56db87b6542bfaa31d2a10412e4be: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.010s
I20260810 03:25:55.867988 31684 raft_consensus.cc:348] T 0bd9248b18c74e949e230c57b0ba90a8 P d9f56db87b6542bfaa31d2a10412e4be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2831154de99842959a694f57c980b448" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 33653 } } peers { permanent_uuid: "d9f56db87b6542bfaa31d2a10412e4be" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 33631 } } peers { permanent_uuid: "a16f2984931649a2a594bdd90b4debae" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 42247 } }
I20260810 03:25:55.868320 31684 raft_consensus.cc:374] T 0bd9248b18c74e949e230c57b0ba90a8 P d9f56db87b6542bfaa31d2a10412e4be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:25:55.868412 31684 raft_consensus.cc:729] T 0bd9248b18c74e949e230c57b0ba90a8 P d9f56db87b6542bfaa31d2a10412e4be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9f56db87b6542bfaa31d2a10412e4be, State: Initialized, Role: FOLLOWER
I20260810 03:25:55.868733 31684 consensus_queue.cc:260] T 0bd9248b18c74e949e230c57b0ba90a8 P d9f56db87b6542bfaa31d2a10412e4be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2831154de99842959a694f57c980b448" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 33653 } } peers { permanent_uuid: "d9f56db87b6542bfaa31d2a10412e4be" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 33631 } } peers { permanent_uuid: "a16f2984931649a2a594bdd90b4debae" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 42247 } }
I20260810 03:25:55.872491 31684 ts_tablet_manager.cc:1428] T 0bd9248b18c74e949e230c57b0ba90a8 P d9f56db87b6542bfaa31d2a10412e4be: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20260810 03:25:55.876961 31687 tablet_bootstrap.cc:492] T 0bd9248b18c74e949e230c57b0ba90a8 P a16f2984931649a2a594bdd90b4debae: No bootstrap required, opened a new log
I20260810 03:25:55.876999 31688 tablet_bootstrap.cc:492] T 0bd9248b18c74e949e230c57b0ba90a8 P 2831154de99842959a694f57c980b448: No bootstrap required, opened a new log
I20260810 03:25:55.877183 31688 ts_tablet_manager.cc:1397] T 0bd9248b18c74e949e230c57b0ba90a8 P 2831154de99842959a694f57c980b448: Time spent bootstrapping tablet: real 0.022s user 0.007s sys 0.013s
I20260810 03:25:55.877235 31687 ts_tablet_manager.cc:1397] T 0bd9248b18c74e949e230c57b0ba90a8 P a16f2984931649a2a594bdd90b4debae: Time spent bootstrapping tablet: real 0.025s user 0.016s sys 0.008s
I20260810 03:25:55.878039 31688 raft_consensus.cc:348] T 0bd9248b18c74e949e230c57b0ba90a8 P 2831154de99842959a694f57c980b448 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2831154de99842959a694f57c980b448" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 33653 } } peers { permanent_uuid: "d9f56db87b6542bfaa31d2a10412e4be" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 33631 } } peers { permanent_uuid: "a16f2984931649a2a594bdd90b4debae" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 42247 } }
I20260810 03:25:55.878124 31687 raft_consensus.cc:348] T 0bd9248b18c74e949e230c57b0ba90a8 P a16f2984931649a2a594bdd90b4debae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2831154de99842959a694f57c980b448" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 33653 } } peers { permanent_uuid: "d9f56db87b6542bfaa31d2a10412e4be" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 33631 } } peers { permanent_uuid: "a16f2984931649a2a594bdd90b4debae" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 42247 } }
I20260810 03:25:55.878320 31688 raft_consensus.cc:374] T 0bd9248b18c74e949e230c57b0ba90a8 P 2831154de99842959a694f57c980b448 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:25:55.878371 31687 raft_consensus.cc:374] T 0bd9248b18c74e949e230c57b0ba90a8 P a16f2984931649a2a594bdd90b4debae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:25:55.878401 31688 raft_consensus.cc:729] T 0bd9248b18c74e949e230c57b0ba90a8 P 2831154de99842959a694f57c980b448 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2831154de99842959a694f57c980b448, State: Initialized, Role: FOLLOWER
I20260810 03:25:55.878448 31687 raft_consensus.cc:729] T 0bd9248b18c74e949e230c57b0ba90a8 P a16f2984931649a2a594bdd90b4debae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a16f2984931649a2a594bdd90b4debae, State: Initialized, Role: FOLLOWER
I20260810 03:25:55.878683 31688 consensus_queue.cc:260] T 0bd9248b18c74e949e230c57b0ba90a8 P 2831154de99842959a694f57c980b448 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2831154de99842959a694f57c980b448" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 33653 } } peers { permanent_uuid: "d9f56db87b6542bfaa31d2a10412e4be" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 33631 } } peers { permanent_uuid: "a16f2984931649a2a594bdd90b4debae" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 42247 } }
I20260810 03:25:55.878706 31687 consensus_queue.cc:260] T 0bd9248b18c74e949e230c57b0ba90a8 P a16f2984931649a2a594bdd90b4debae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2831154de99842959a694f57c980b448" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 33653 } } peers { permanent_uuid: "d9f56db87b6542bfaa31d2a10412e4be" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 33631 } } peers { permanent_uuid: "a16f2984931649a2a594bdd90b4debae" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 42247 } }
I20260810 03:25:55.879290 31688 ts_tablet_manager.cc:1428] T 0bd9248b18c74e949e230c57b0ba90a8 P 2831154de99842959a694f57c980b448: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260810 03:25:55.879598 31687 ts_tablet_manager.cc:1428] T 0bd9248b18c74e949e230c57b0ba90a8 P a16f2984931649a2a594bdd90b4debae: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260810 03:25:55.887499 31693 raft_consensus.cc:482] T 0bd9248b18c74e949e230c57b0ba90a8 P d9f56db87b6542bfaa31d2a10412e4be [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 03:25:55.887722 31693 raft_consensus.cc:504] T 0bd9248b18c74e949e230c57b0ba90a8 P d9f56db87b6542bfaa31d2a10412e4be [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2831154de99842959a694f57c980b448" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 33653 } } peers { permanent_uuid: "d9f56db87b6542bfaa31d2a10412e4be" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 33631 } } peers { permanent_uuid: "a16f2984931649a2a594bdd90b4debae" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 42247 } }
I20260810 03:25:55.889231 31693 leader_election.cc:288] T 0bd9248b18c74e949e230c57b0ba90a8 P d9f56db87b6542bfaa31d2a10412e4be [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2831154de99842959a694f57c980b448 (127.27.161.1:33653), a16f2984931649a2a594bdd90b4debae (127.27.161.3:42247)
I20260810 03:25:55.930431 31765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:55.889409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.1:33653 (local address 127.27.161.2:55869)
0810 03:25:55.922418 (+ 33009us) negotiation.cc:107] Waiting for socket to connect
0810 03:25:55.922444 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0810 03:25:55.922645 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:25:55.923163 (+ 518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:25:55.923168 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:25:55.923201 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:25:55.923540 (+ 339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:25:55.923553 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:25:55.925337 (+ 1784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:25:55.925343 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:25:55.927619 (+ 2276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:25:55.927636 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:25:55.929474 (+ 1838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:25:55.929480 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:25:55.929657 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:25:55.929690 (+ 33us) client_negotiation.cc:769] Sending connection context
0810 03:25:55.929787 (+ 97us) client_negotiation.cc:240] Negotiation successful
0810 03:25:55.929971 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32859,"mutex_wait_us":40,"thread_start_us":32723,"threads_started":1}
I20260810 03:25:55.930784 31721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:55.891234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.161.2:53649 (local address 127.27.161.3:42247)
0810 03:25:55.891418 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0810 03:25:55.891424 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:25:55.923362 (+ 31938us) server_negotiation.cc:407] Connection header received
0810 03:25:55.923517 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:25:55.923521 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:25:55.923603 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:25:55.923702 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:25:55.924471 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:55.925483 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:55.928149 (+ 2666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:55.929521 (+ 1372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:55.929649 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:25:55.929722 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0810 03:25:55.930346 (+ 624us) server_negotiation.cc:299] Negotiation successful
0810 03:25:55.930491 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260810 03:25:55.931186 31775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:55.890383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.161.2:55869 (local address 127.27.161.1:33653)
0810 03:25:55.913640 (+ 23257us) server_negotiation.cc:206] Beginning negotiation
0810 03:25:55.913647 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:25:55.922600 (+ 8953us) server_negotiation.cc:407] Connection header received
0810 03:25:55.922792 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:25:55.922798 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:25:55.922907 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:25:55.923020 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:25:55.923739 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:55.925158 (+ 1419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:55.927806 (+ 2648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:55.929079 (+ 1273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:55.929232 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:25:55.929328 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0810 03:25:55.930693 (+ 1365us) server_negotiation.cc:299] Negotiation successful
0810 03:25:55.930845 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23101,"thread_start_us":23263,"threads_started":1}
I20260810 03:25:55.931497 31766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:55.889658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.3:42247 (local address 127.27.161.2:53649)
0810 03:25:55.923211 (+ 33553us) negotiation.cc:107] Waiting for socket to connect
0810 03:25:55.923247 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0810 03:25:55.923393 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:25:55.923861 (+ 468us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:25:55.923866 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:25:55.923891 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:25:55.924263 (+ 372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:25:55.924275 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:25:55.925684 (+ 1409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:25:55.925689 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:25:55.927991 (+ 2302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:25:55.928005 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:25:55.929925 (+ 1920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:25:55.929930 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:25:55.930139 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:25:55.930175 (+ 36us) client_negotiation.cc:769] Sending connection context
0810 03:25:55.930967 (+ 792us) client_negotiation.cc:240] Negotiation successful
0810 03:25:55.931112 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33413,"mutex_wait_us":29,"thread_start_us":33224,"threads_started":1}
I20260810 03:25:55.932042 31151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bd9248b18c74e949e230c57b0ba90a8" candidate_uuid: "d9f56db87b6542bfaa31d2a10412e4be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2831154de99842959a694f57c980b448" is_pre_election: true
I20260810 03:25:55.932335 31151 raft_consensus.cc:2454] T 0bd9248b18c74e949e230c57b0ba90a8 P 2831154de99842959a694f57c980b448 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9f56db87b6542bfaa31d2a10412e4be in term 0.
I20260810 03:25:55.932472 31590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bd9248b18c74e949e230c57b0ba90a8" candidate_uuid: "d9f56db87b6542bfaa31d2a10412e4be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a16f2984931649a2a594bdd90b4debae" is_pre_election: true
I20260810 03:25:55.932953 31590 raft_consensus.cc:2454] T 0bd9248b18c74e949e230c57b0ba90a8 P a16f2984931649a2a594bdd90b4debae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9f56db87b6542bfaa31d2a10412e4be in term 0.
I20260810 03:25:55.932971 31255 leader_election.cc:302] T 0bd9248b18c74e949e230c57b0ba90a8 P d9f56db87b6542bfaa31d2a10412e4be [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2831154de99842959a694f57c980b448, d9f56db87b6542bfaa31d2a10412e4be; no voters:
I20260810 03:25:55.933610 31693 raft_consensus.cc:2781] T 0bd9248b18c74e949e230c57b0ba90a8 P d9f56db87b6542bfaa31d2a10412e4be [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 03:25:55.933758 31693 raft_consensus.cc:482] T 0bd9248b18c74e949e230c57b0ba90a8 P d9f56db87b6542bfaa31d2a10412e4be [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 03:25:55.933864 31693 raft_consensus.cc:3037] T 0bd9248b18c74e949e230c57b0ba90a8 P d9f56db87b6542bfaa31d2a10412e4be [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:25:55.936499 31693 raft_consensus.cc:504] T 0bd9248b18c74e949e230c57b0ba90a8 P d9f56db87b6542bfaa31d2a10412e4be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2831154de99842959a694f57c980b448" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 33653 } } peers { permanent_uuid: "d9f56db87b6542bfaa31d2a10412e4be" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 33631 } } peers { permanent_uuid: "a16f2984931649a2a594bdd90b4debae" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 42247 } }
I20260810 03:25:55.937232 31693 leader_election.cc:288] T 0bd9248b18c74e949e230c57b0ba90a8 P d9f56db87b6542bfaa31d2a10412e4be [CANDIDATE]: Term 1 election: Requested vote from peers 2831154de99842959a694f57c980b448 (127.27.161.1:33653), a16f2984931649a2a594bdd90b4debae (127.27.161.3:42247)
I20260810 03:25:55.937758 31151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bd9248b18c74e949e230c57b0ba90a8" candidate_uuid: "d9f56db87b6542bfaa31d2a10412e4be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2831154de99842959a694f57c980b448"
I20260810 03:25:55.938020 31151 raft_consensus.cc:3037] T 0bd9248b18c74e949e230c57b0ba90a8 P 2831154de99842959a694f57c980b448 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:25:55.937922 31590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bd9248b18c74e949e230c57b0ba90a8" candidate_uuid: "d9f56db87b6542bfaa31d2a10412e4be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a16f2984931649a2a594bdd90b4debae"
I20260810 03:25:55.938153 31590 raft_consensus.cc:3037] T 0bd9248b18c74e949e230c57b0ba90a8 P a16f2984931649a2a594bdd90b4debae [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:25:55.941314 31151 raft_consensus.cc:2454] T 0bd9248b18c74e949e230c57b0ba90a8 P 2831154de99842959a694f57c980b448 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9f56db87b6542bfaa31d2a10412e4be in term 1.
I20260810 03:25:55.941314 31590 raft_consensus.cc:2454] T 0bd9248b18c74e949e230c57b0ba90a8 P a16f2984931649a2a594bdd90b4debae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9f56db87b6542bfaa31d2a10412e4be in term 1.
I20260810 03:25:55.941861 31256 leader_election.cc:302] T 0bd9248b18c74e949e230c57b0ba90a8 P d9f56db87b6542bfaa31d2a10412e4be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a16f2984931649a2a594bdd90b4debae, d9f56db87b6542bfaa31d2a10412e4be; no voters:
I20260810 03:25:55.942258 31693 raft_consensus.cc:2781] T 0bd9248b18c74e949e230c57b0ba90a8 P d9f56db87b6542bfaa31d2a10412e4be [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:25:55.942788 31693 raft_consensus.cc:686] T 0bd9248b18c74e949e230c57b0ba90a8 P d9f56db87b6542bfaa31d2a10412e4be [term 1 LEADER]: Becoming Leader. State: Replica: d9f56db87b6542bfaa31d2a10412e4be, State: Running, Role: LEADER
I20260810 03:25:55.943269 31693 consensus_queue.cc:237] T 0bd9248b18c74e949e230c57b0ba90a8 P d9f56db87b6542bfaa31d2a10412e4be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2831154de99842959a694f57c980b448" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 33653 } } peers { permanent_uuid: "d9f56db87b6542bfaa31d2a10412e4be" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 33631 } } peers { permanent_uuid: "a16f2984931649a2a594bdd90b4debae" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 42247 } }
I20260810 03:25:55.947336 30906 catalog_manager.cc:5495] T 0bd9248b18c74e949e230c57b0ba90a8 P d9f56db87b6542bfaa31d2a10412e4be reported cstate change: term changed from 0 to 1, leader changed from <none> to d9f56db87b6542bfaa31d2a10412e4be (127.27.161.2). New cstate: current_term: 1 leader_uuid: "d9f56db87b6542bfaa31d2a10412e4be" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2831154de99842959a694f57c980b448" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 33653 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d9f56db87b6542bfaa31d2a10412e4be" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 33631 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a16f2984931649a2a594bdd90b4debae" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 42247 } health_report { overall_health: UNKNOWN } } }
I20260810 03:25:55.997598 31780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:55.986744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40126 (local address 127.27.161.2:33631)
0810 03:25:55.990495 (+ 3751us) server_negotiation.cc:206] Beginning negotiation
0810 03:25:55.990501 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:25:55.990533 (+ 32us) server_negotiation.cc:407] Connection header received
0810 03:25:55.990658 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:25:55.990664 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:25:55.990786 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:25:55.991018 (+ 232us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 03:25:55.991459 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:55.992660 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:55.994310 (+ 1650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:55.995110 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:55.995267 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:25:55.995809 (+ 542us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 03:25:55.996689 (+ 880us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 03:25:55.996852 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0810 03:25:55.997083 (+ 231us) server_negotiation.cc:299] Negotiation successful
0810 03:25:55.997192 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3575,"thread_start_us":3435,"threads_started":1}
I20260810 03:25:56.018651 31721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:56.011198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.161.2:38507 (local address 127.27.161.3:42247)
0810 03:25:56.011421 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0810 03:25:56.011428 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:25:56.012151 (+ 723us) server_negotiation.cc:407] Connection header received
0810 03:25:56.012391 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:25:56.012399 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:25:56.012484 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:25:56.012574 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 03:25:56.013181 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:56.014042 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:56.016116 (+ 2074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:56.016846 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:56.016978 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:25:56.017661 (+ 683us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 03:25:56.017865 (+ 204us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 03:25:56.018003 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0810 03:25:56.018158 (+ 155us) server_negotiation.cc:299] Negotiation successful
0810 03:25:56.018268 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260810 03:25:56.018697 31783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:56.011181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.3:42247 (local address 127.27.161.2:38507)
0810 03:25:56.012016 (+ 835us) negotiation.cc:107] Waiting for socket to connect
0810 03:25:56.012038 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0810 03:25:56.012222 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:25:56.012629 (+ 407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:25:56.012634 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:25:56.012648 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 03:25:56.013009 (+ 361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:25:56.013022 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:25:56.014202 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:25:56.014209 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:25:56.015945 (+ 1736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:25:56.015965 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:25:56.017083 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:25:56.017091 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:25:56.017313 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:25:56.017410 (+ 97us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 03:25:56.018008 (+ 598us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 03:25:56.018023 (+ 15us) client_negotiation.cc:769] Sending connection context
0810 03:25:56.018119 (+ 96us) client_negotiation.cc:240] Negotiation successful
0810 03:25:56.018291 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":598,"thread_start_us":370,"threads_started":1}
I20260810 03:25:56.029574 31151 raft_consensus.cc:1261] T 0bd9248b18c74e949e230c57b0ba90a8 P 2831154de99842959a694f57c980b448 [term 1 FOLLOWER]: Refusing update from remote peer d9f56db87b6542bfaa31d2a10412e4be: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:25:56.030548 31590 raft_consensus.cc:1261] T 0bd9248b18c74e949e230c57b0ba90a8 P a16f2984931649a2a594bdd90b4debae [term 1 FOLLOWER]: Refusing update from remote peer d9f56db87b6542bfaa31d2a10412e4be: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:25:56.031504 31776 consensus_queue.cc:1035] T 0bd9248b18c74e949e230c57b0ba90a8 P d9f56db87b6542bfaa31d2a10412e4be [LEADER]: Connected to new peer: Peer: permanent_uuid: "a16f2984931649a2a594bdd90b4debae" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 42247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:25:56.032903 31776 consensus_queue.cc:1035] T 0bd9248b18c74e949e230c57b0ba90a8 P d9f56db87b6542bfaa31d2a10412e4be [LEADER]: Connected to new peer: Peer: permanent_uuid: "2831154de99842959a694f57c980b448" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 33653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:25:56.085513 31748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:56.077064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.2:33631 (local address 127.27.161.3:41187)
0810 03:25:56.078085 (+ 1021us) negotiation.cc:107] Waiting for socket to connect
0810 03:25:56.078121 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0810 03:25:56.078366 (+ 245us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:25:56.079883 (+ 1517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:25:56.079889 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:25:56.079907 (+ 18us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 03:25:56.080199 (+ 292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:25:56.080215 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:25:56.081783 (+ 1568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:25:56.081791 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:25:56.083298 (+ 1507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:25:56.083320 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:25:56.084195 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:25:56.084201 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:25:56.084427 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:25:56.084509 (+ 82us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 03:25:56.084965 (+ 456us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 03:25:56.084978 (+ 13us) client_negotiation.cc:769] Sending connection context
0810 03:25:56.085064 (+ 86us) client_negotiation.cc:240] Negotiation successful
0810 03:25:56.085204 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129}
I20260810 03:25:56.086205 31780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:56.079250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.161.3:41187 (local address 127.27.161.2:33631)
0810 03:25:56.079462 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0810 03:25:56.079469 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:25:56.079493 (+ 24us) server_negotiation.cc:407] Connection header received
0810 03:25:56.079576 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:25:56.079582 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:25:56.079677 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:25:56.079803 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 03:25:56.080392 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:56.081588 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:56.083522 (+ 1934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:56.084031 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:56.084179 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:25:56.084717 (+ 538us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 03:25:56.084862 (+ 145us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 03:25:56.084954 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0810 03:25:56.085766 (+ 812us) server_negotiation.cc:299] Negotiation successful
0810 03:25:56.085859 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
W20260810 03:25:56.087379 31723 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260810 03:25:57.729282 30906 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55662:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 03:25:57.729601 30906 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55662:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 03:25:57.734754 30906 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0733cfa08dc743d8a587b321ff9eba88: Sending DeleteTablet for 3 replicas of tablet 0bd9248b18c74e949e230c57b0ba90a8
I20260810 03:25:57.735754 31126 tablet_service.cc:1514] Processing DeleteTablet for tablet 0bd9248b18c74e949e230c57b0ba90a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:25:57 UTC) from {username='kudu', principal='kudu/127.27.161.62@KRBTEST.COM'} at 127.0.0.1:59472
I20260810 03:25:57.735776 31559 tablet_service.cc:1514] Processing DeleteTablet for tablet 0bd9248b18c74e949e230c57b0ba90a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:25:57 UTC) from {username='kudu', principal='kudu/127.27.161.62@KRBTEST.COM'} at 127.0.0.1:44192
I20260810 03:25:57.735883 31331 tablet_service.cc:1514] Processing DeleteTablet for tablet 0bd9248b18c74e949e230c57b0ba90a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:25:57 UTC) from {username='kudu', principal='kudu/127.27.161.62@KRBTEST.COM'} at 127.0.0.1:40112
I20260810 03:25:57.748137 31826 tablet_replica.cc:330] T 0bd9248b18c74e949e230c57b0ba90a8 P 2831154de99842959a694f57c980b448: stopping tablet replica
I20260810 03:25:57.748384 31829 tablet_replica.cc:330] T 0bd9248b18c74e949e230c57b0ba90a8 P d9f56db87b6542bfaa31d2a10412e4be: stopping tablet replica
I20260810 03:25:57.748404 31826 raft_consensus.cc:2229] T 0bd9248b18c74e949e230c57b0ba90a8 P 2831154de99842959a694f57c980b448 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 03:25:57.748591 31829 raft_consensus.cc:2229] T 0bd9248b18c74e949e230c57b0ba90a8 P d9f56db87b6542bfaa31d2a10412e4be [term 1 LEADER]: Raft consensus shutting down.
I20260810 03:25:57.748677 31826 raft_consensus.cc:2258] T 0bd9248b18c74e949e230c57b0ba90a8 P 2831154de99842959a694f57c980b448 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:25:57.749346 31829 raft_consensus.cc:2258] T 0bd9248b18c74e949e230c57b0ba90a8 P d9f56db87b6542bfaa31d2a10412e4be [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:25:57.750489 31826 ts_tablet_manager.cc:1905] T 0bd9248b18c74e949e230c57b0ba90a8 P 2831154de99842959a694f57c980b448: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:25:57.750546 31829 ts_tablet_manager.cc:1905] T 0bd9248b18c74e949e230c57b0ba90a8 P d9f56db87b6542bfaa31d2a10412e4be: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:25:57.751325 31827 tablet_replica.cc:330] T 0bd9248b18c74e949e230c57b0ba90a8 P a16f2984931649a2a594bdd90b4debae: stopping tablet replica
I20260810 03:25:57.751554 31827 raft_consensus.cc:2229] T 0bd9248b18c74e949e230c57b0ba90a8 P a16f2984931649a2a594bdd90b4debae [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 03:25:57.751791 31827 raft_consensus.cc:2258] T 0bd9248b18c74e949e230c57b0ba90a8 P a16f2984931649a2a594bdd90b4debae [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:25:57.752796 31827 ts_tablet_manager.cc:1905] T 0bd9248b18c74e949e230c57b0ba90a8 P a16f2984931649a2a594bdd90b4debae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:25:57.757419 31829 ts_tablet_manager.cc:1918] T 0bd9248b18c74e949e230c57b0ba90a8 P d9f56db87b6542bfaa31d2a10412e4be: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260810 03:25:57.757485 31826 ts_tablet_manager.cc:1918] T 0bd9248b18c74e949e230c57b0ba90a8 P 2831154de99842959a694f57c980b448: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260810 03:25:57.757611 31826 log.cc:1196] T 0bd9248b18c74e949e230c57b0ba90a8 P 2831154de99842959a694f57c980b448: Deleting WAL directory at /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786332336319544-28292-0/minicluster-data/ts-0/wal/wals/0bd9248b18c74e949e230c57b0ba90a8
I20260810 03:25:57.757673 31829 log.cc:1196] T 0bd9248b18c74e949e230c57b0ba90a8 P d9f56db87b6542bfaa31d2a10412e4be: Deleting WAL directory at /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786332336319544-28292-0/minicluster-data/ts-1/wal/wals/0bd9248b18c74e949e230c57b0ba90a8
I20260810 03:25:57.758208 31826 ts_tablet_manager.cc:1939] T 0bd9248b18c74e949e230c57b0ba90a8 P 2831154de99842959a694f57c980b448: Deleting consensus metadata
I20260810 03:25:57.758219 31829 ts_tablet_manager.cc:1939] T 0bd9248b18c74e949e230c57b0ba90a8 P d9f56db87b6542bfaa31d2a10412e4be: Deleting consensus metadata
I20260810 03:25:57.759557 30887 catalog_manager.cc:4841] TS 2831154de99842959a694f57c980b448 (127.27.161.1:33653): tablet 0bd9248b18c74e949e230c57b0ba90a8 (table test-table [id=0b7e97ce95f74a95864447818102993b]) successfully deleted
I20260810 03:25:57.759557 30888 catalog_manager.cc:4841] TS d9f56db87b6542bfaa31d2a10412e4be (127.27.161.2:33631): tablet 0bd9248b18c74e949e230c57b0ba90a8 (table test-table [id=0b7e97ce95f74a95864447818102993b]) successfully deleted
I20260810 03:25:57.759928 31827 ts_tablet_manager.cc:1918] T 0bd9248b18c74e949e230c57b0ba90a8 P a16f2984931649a2a594bdd90b4debae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260810 03:25:57.760109 31827 log.cc:1196] T 0bd9248b18c74e949e230c57b0ba90a8 P a16f2984931649a2a594bdd90b4debae: Deleting WAL directory at /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786332336319544-28292-0/minicluster-data/ts-2/wal/wals/0bd9248b18c74e949e230c57b0ba90a8
I20260810 03:25:57.760584 31827 ts_tablet_manager.cc:1939] T 0bd9248b18c74e949e230c57b0ba90a8 P a16f2984931649a2a594bdd90b4debae: Deleting consensus metadata
I20260810 03:25:57.761879 30888 catalog_manager.cc:4841] TS a16f2984931649a2a594bdd90b4debae (127.27.161.3:42247): tablet 0bd9248b18c74e949e230c57b0ba90a8 (table test-table [id=0b7e97ce95f74a95864447818102993b]) successfully deleted
Aug 10 03:25:57 dist-test-slave-vpvm krb5kdc[30840](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332357, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 10 03:25:57 dist-test-slave-vpvm krb5kdc[30840](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332357, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.27.161.62@KRBTEST.COM
I20260810 03:25:57.806953 31839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:57.792588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55680 (local address 127.27.161.62:38085)
0810 03:25:57.793176 (+ 588us) server_negotiation.cc:206] Beginning negotiation
0810 03:25:57.793181 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:25:57.793205 (+ 24us) server_negotiation.cc:407] Connection header received
0810 03:25:57.793301 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:25:57.793306 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:25:57.793369 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:25:57.793463 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:25:57.794911 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:57.795731 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:57.797067 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:57.797457 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:57.797676 (+ 219us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:25:57.801625 (+ 3949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:25:57.801631 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:25:57.801634 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:25:57.801675 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:25:57.804504 (+ 2829us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:25:57.805135 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:25:57.805138 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:25:57.805141 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:25:57.805292 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:25:57.805587 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:25:57.805591 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:25:57.805593 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:25:57.805876 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:25:57.806111 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0810 03:25:57.806435 (+ 324us) server_negotiation.cc:299] Negotiation successful
0810 03:25:57.806632 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":472,"thread_start_us":285,"threads_started":1}
I20260810 03:25:57.810766 30906 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55680:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 03:25:57.811374 30906 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 03:25:57.818002 31126 tablet_service.cc:1467] Processing CreateTablet for tablet 6e89196258634e11ae52fa1d0cd62c79 (DEFAULT_TABLE table=test-table [id=fc0fb98a4b9e464daa1ec0bdf4452948]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:25:57.818197 31331 tablet_service.cc:1467] Processing CreateTablet for tablet 6e89196258634e11ae52fa1d0cd62c79 (DEFAULT_TABLE table=test-table [id=fc0fb98a4b9e464daa1ec0bdf4452948]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:25:57.818512 31126 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e89196258634e11ae52fa1d0cd62c79. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:25:57.818435 31559 tablet_service.cc:1467] Processing CreateTablet for tablet 6e89196258634e11ae52fa1d0cd62c79 (DEFAULT_TABLE table=test-table [id=fc0fb98a4b9e464daa1ec0bdf4452948]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:25:57.818683 31331 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e89196258634e11ae52fa1d0cd62c79. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:25:57.818930 31559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e89196258634e11ae52fa1d0cd62c79. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:25:57.824725 31840 tablet_bootstrap.cc:492] T 6e89196258634e11ae52fa1d0cd62c79 P 2831154de99842959a694f57c980b448: Bootstrap starting.
I20260810 03:25:57.825577 31842 tablet_bootstrap.cc:492] T 6e89196258634e11ae52fa1d0cd62c79 P a16f2984931649a2a594bdd90b4debae: Bootstrap starting.
I20260810 03:25:57.826536 31843 tablet_bootstrap.cc:492] T 6e89196258634e11ae52fa1d0cd62c79 P d9f56db87b6542bfaa31d2a10412e4be: Bootstrap starting.
I20260810 03:25:57.826638 31840 tablet_bootstrap.cc:654] T 6e89196258634e11ae52fa1d0cd62c79 P 2831154de99842959a694f57c980b448: Neither blocks nor log segments found. Creating new log.
I20260810 03:25:57.827395 31842 tablet_bootstrap.cc:654] T 6e89196258634e11ae52fa1d0cd62c79 P a16f2984931649a2a594bdd90b4debae: Neither blocks nor log segments found. Creating new log.
I20260810 03:25:57.828505 31843 tablet_bootstrap.cc:654] T 6e89196258634e11ae52fa1d0cd62c79 P d9f56db87b6542bfaa31d2a10412e4be: Neither blocks nor log segments found. Creating new log.
I20260810 03:25:57.829627 31840 tablet_bootstrap.cc:492] T 6e89196258634e11ae52fa1d0cd62c79 P 2831154de99842959a694f57c980b448: No bootstrap required, opened a new log
I20260810 03:25:57.829784 31840 ts_tablet_manager.cc:1397] T 6e89196258634e11ae52fa1d0cd62c79 P 2831154de99842959a694f57c980b448: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260810 03:25:57.830744 31843 tablet_bootstrap.cc:492] T 6e89196258634e11ae52fa1d0cd62c79 P d9f56db87b6542bfaa31d2a10412e4be: No bootstrap required, opened a new log
I20260810 03:25:57.830849 31843 ts_tablet_manager.cc:1397] T 6e89196258634e11ae52fa1d0cd62c79 P d9f56db87b6542bfaa31d2a10412e4be: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260810 03:25:57.830667 31840 raft_consensus.cc:348] T 6e89196258634e11ae52fa1d0cd62c79 P 2831154de99842959a694f57c980b448 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2831154de99842959a694f57c980b448" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 33653 } } peers { permanent_uuid: "d9f56db87b6542bfaa31d2a10412e4be" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 33631 } } peers { permanent_uuid: "a16f2984931649a2a594bdd90b4debae" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 42247 } }
I20260810 03:25:57.830968 31840 raft_consensus.cc:374] T 6e89196258634e11ae52fa1d0cd62c79 P 2831154de99842959a694f57c980b448 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:25:57.831089 31840 raft_consensus.cc:729] T 6e89196258634e11ae52fa1d0cd62c79 P 2831154de99842959a694f57c980b448 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2831154de99842959a694f57c980b448, State: Initialized, Role: FOLLOWER
I20260810 03:25:57.831365 31840 consensus_queue.cc:260] T 6e89196258634e11ae52fa1d0cd62c79 P 2831154de99842959a694f57c980b448 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2831154de99842959a694f57c980b448" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 33653 } } peers { permanent_uuid: "d9f56db87b6542bfaa31d2a10412e4be" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 33631 } } peers { permanent_uuid: "a16f2984931649a2a594bdd90b4debae" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 42247 } }
I20260810 03:25:57.831637 31843 raft_consensus.cc:348] T 6e89196258634e11ae52fa1d0cd62c79 P d9f56db87b6542bfaa31d2a10412e4be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2831154de99842959a694f57c980b448" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 33653 } } peers { permanent_uuid: "d9f56db87b6542bfaa31d2a10412e4be" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 33631 } } peers { permanent_uuid: "a16f2984931649a2a594bdd90b4debae" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 42247 } }
I20260810 03:25:57.831913 31843 raft_consensus.cc:374] T 6e89196258634e11ae52fa1d0cd62c79 P d9f56db87b6542bfaa31d2a10412e4be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:25:57.832083 31843 raft_consensus.cc:729] T 6e89196258634e11ae52fa1d0cd62c79 P d9f56db87b6542bfaa31d2a10412e4be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9f56db87b6542bfaa31d2a10412e4be, State: Initialized, Role: FOLLOWER
I20260810 03:25:57.832440 31843 consensus_queue.cc:260] T 6e89196258634e11ae52fa1d0cd62c79 P d9f56db87b6542bfaa31d2a10412e4be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2831154de99842959a694f57c980b448" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 33653 } } peers { permanent_uuid: "d9f56db87b6542bfaa31d2a10412e4be" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 33631 } } peers { permanent_uuid: "a16f2984931649a2a594bdd90b4debae" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 42247 } }
I20260810 03:25:57.833089 31840 ts_tablet_manager.cc:1428] T 6e89196258634e11ae52fa1d0cd62c79 P 2831154de99842959a694f57c980b448: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260810 03:25:57.833572 31842 tablet_bootstrap.cc:492] T 6e89196258634e11ae52fa1d0cd62c79 P a16f2984931649a2a594bdd90b4debae: No bootstrap required, opened a new log
I20260810 03:25:57.833725 31843 ts_tablet_manager.cc:1428] T 6e89196258634e11ae52fa1d0cd62c79 P d9f56db87b6542bfaa31d2a10412e4be: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260810 03:25:57.833772 31842 ts_tablet_manager.cc:1397] T 6e89196258634e11ae52fa1d0cd62c79 P a16f2984931649a2a594bdd90b4debae: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.004s
I20260810 03:25:57.834808 31842 raft_consensus.cc:348] T 6e89196258634e11ae52fa1d0cd62c79 P a16f2984931649a2a594bdd90b4debae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2831154de99842959a694f57c980b448" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 33653 } } peers { permanent_uuid: "d9f56db87b6542bfaa31d2a10412e4be" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 33631 } } peers { permanent_uuid: "a16f2984931649a2a594bdd90b4debae" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 42247 } }
I20260810 03:25:57.835083 31842 raft_consensus.cc:374] T 6e89196258634e11ae52fa1d0cd62c79 P a16f2984931649a2a594bdd90b4debae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:25:57.835196 31842 raft_consensus.cc:729] T 6e89196258634e11ae52fa1d0cd62c79 P a16f2984931649a2a594bdd90b4debae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a16f2984931649a2a594bdd90b4debae, State: Initialized, Role: FOLLOWER
I20260810 03:25:57.835453 31842 consensus_queue.cc:260] T 6e89196258634e11ae52fa1d0cd62c79 P a16f2984931649a2a594bdd90b4debae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2831154de99842959a694f57c980b448" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 33653 } } peers { permanent_uuid: "d9f56db87b6542bfaa31d2a10412e4be" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 33631 } } peers { permanent_uuid: "a16f2984931649a2a594bdd90b4debae" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 42247 } }
I20260810 03:25:57.836103 31842 ts_tablet_manager.cc:1428] T 6e89196258634e11ae52fa1d0cd62c79 P a16f2984931649a2a594bdd90b4debae: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260810 03:25:57.882812 31848 raft_consensus.cc:482] T 6e89196258634e11ae52fa1d0cd62c79 P 2831154de99842959a694f57c980b448 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 03:25:57.883073 31848 raft_consensus.cc:504] T 6e89196258634e11ae52fa1d0cd62c79 P 2831154de99842959a694f57c980b448 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2831154de99842959a694f57c980b448" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 33653 } } peers { permanent_uuid: "d9f56db87b6542bfaa31d2a10412e4be" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 33631 } } peers { permanent_uuid: "a16f2984931649a2a594bdd90b4debae" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 42247 } }
I20260810 03:25:57.884694 31848 leader_election.cc:288] T 6e89196258634e11ae52fa1d0cd62c79 P 2831154de99842959a694f57c980b448 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d9f56db87b6542bfaa31d2a10412e4be (127.27.161.2:33631), a16f2984931649a2a594bdd90b4debae (127.27.161.3:42247)
I20260810 03:25:57.892505 31850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:57.885086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.161.1:40615 (local address 127.27.161.2:33631)
0810 03:25:57.885737 (+ 651us) server_negotiation.cc:206] Beginning negotiation
0810 03:25:57.885743 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:25:57.885769 (+ 26us) server_negotiation.cc:407] Connection header received
0810 03:25:57.886089 (+ 320us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:25:57.886095 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:25:57.886196 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:25:57.886859 (+ 663us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:25:57.887058 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:57.887871 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:57.890291 (+ 2420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:57.891459 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:57.891663 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:25:57.891748 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0810 03:25:57.892032 (+ 284us) server_negotiation.cc:299] Negotiation successful
0810 03:25:57.892176 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":508,"thread_start_us":369,"threads_started":1}
I20260810 03:25:57.892549 31849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:57.884969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.2:33631 (local address 127.27.161.1:40615)
0810 03:25:57.885679 (+ 710us) negotiation.cc:107] Waiting for socket to connect
0810 03:25:57.885701 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0810 03:25:57.885901 (+ 200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:25:57.886369 (+ 468us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:25:57.886375 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:25:57.886425 (+ 50us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:25:57.886798 (+ 373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:25:57.886812 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:25:57.887999 (+ 1187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:25:57.888005 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:25:57.890091 (+ 2086us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:25:57.890108 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:25:57.891680 (+ 1572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:25:57.891686 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:25:57.891874 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:25:57.891901 (+ 27us) client_negotiation.cc:769] Sending connection context
0810 03:25:57.891995 (+ 94us) client_negotiation.cc:240] Negotiation successful
0810 03:25:57.892138 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":567,"mutex_wait_us":26,"thread_start_us":393,"threads_started":1}
I20260810 03:25:57.893708 31854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:57.885848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.3:42247 (local address 127.27.161.1:35445)
0810 03:25:57.886621 (+ 773us) negotiation.cc:107] Waiting for socket to connect
0810 03:25:57.886638 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 03:25:57.886750 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:25:57.887181 (+ 431us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:25:57.887186 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:25:57.887211 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:25:57.887524 (+ 313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:25:57.887534 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:25:57.888871 (+ 1337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:25:57.888875 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:25:57.890923 (+ 2048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:25:57.890940 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:25:57.892776 (+ 1836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:25:57.892783 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:25:57.892963 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:25:57.892997 (+ 34us) client_negotiation.cc:769] Sending connection context
0810 03:25:57.893083 (+ 86us) client_negotiation.cc:240] Negotiation successful
0810 03:25:57.893235 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":603,"mutex_wait_us":21,"thread_start_us":382,"threads_started":1}
I20260810 03:25:57.893729 31856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:57.886014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.161.1:35445 (local address 127.27.161.3:42247)
0810 03:25:57.886883 (+ 869us) server_negotiation.cc:206] Beginning negotiation
0810 03:25:57.886889 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:25:57.886911 (+ 22us) server_negotiation.cc:407] Connection header received
0810 03:25:57.886984 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:25:57.886989 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:25:57.887055 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:25:57.887138 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:25:57.887687 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:57.888750 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:57.891104 (+ 2354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:57.892330 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:57.892559 (+ 229us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:25:57.892636 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0810 03:25:57.893236 (+ 600us) server_negotiation.cc:299] Negotiation successful
0810 03:25:57.893382 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":698,"thread_start_us":515,"threads_started":1}
I20260810 03:25:57.893603 31358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e89196258634e11ae52fa1d0cd62c79" candidate_uuid: "2831154de99842959a694f57c980b448" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9f56db87b6542bfaa31d2a10412e4be" is_pre_election: true
I20260810 03:25:57.893919 31358 raft_consensus.cc:2454] T 6e89196258634e11ae52fa1d0cd62c79 P d9f56db87b6542bfaa31d2a10412e4be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2831154de99842959a694f57c980b448 in term 0.
I20260810 03:25:57.894619 31590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e89196258634e11ae52fa1d0cd62c79" candidate_uuid: "2831154de99842959a694f57c980b448" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a16f2984931649a2a594bdd90b4debae" is_pre_election: true
I20260810 03:25:57.894734 31045 leader_election.cc:302] T 6e89196258634e11ae52fa1d0cd62c79 P 2831154de99842959a694f57c980b448 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2831154de99842959a694f57c980b448, d9f56db87b6542bfaa31d2a10412e4be; no voters:
I20260810 03:25:57.894896 31590 raft_consensus.cc:2454] T 6e89196258634e11ae52fa1d0cd62c79 P a16f2984931649a2a594bdd90b4debae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2831154de99842959a694f57c980b448 in term 0.
I20260810 03:25:57.895198 31848 raft_consensus.cc:2781] T 6e89196258634e11ae52fa1d0cd62c79 P 2831154de99842959a694f57c980b448 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 03:25:57.895354 31848 raft_consensus.cc:482] T 6e89196258634e11ae52fa1d0cd62c79 P 2831154de99842959a694f57c980b448 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 03:25:57.895485 31848 raft_consensus.cc:3037] T 6e89196258634e11ae52fa1d0cd62c79 P 2831154de99842959a694f57c980b448 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:25:57.898182 31848 raft_consensus.cc:504] T 6e89196258634e11ae52fa1d0cd62c79 P 2831154de99842959a694f57c980b448 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2831154de99842959a694f57c980b448" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 33653 } } peers { permanent_uuid: "d9f56db87b6542bfaa31d2a10412e4be" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 33631 } } peers { permanent_uuid: "a16f2984931649a2a594bdd90b4debae" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 42247 } }
I20260810 03:25:57.898988 31848 leader_election.cc:288] T 6e89196258634e11ae52fa1d0cd62c79 P 2831154de99842959a694f57c980b448 [CANDIDATE]: Term 1 election: Requested vote from peers d9f56db87b6542bfaa31d2a10412e4be (127.27.161.2:33631), a16f2984931649a2a594bdd90b4debae (127.27.161.3:42247)
I20260810 03:25:57.899755 31358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e89196258634e11ae52fa1d0cd62c79" candidate_uuid: "2831154de99842959a694f57c980b448" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9f56db87b6542bfaa31d2a10412e4be"
I20260810 03:25:57.900009 31358 raft_consensus.cc:3037] T 6e89196258634e11ae52fa1d0cd62c79 P d9f56db87b6542bfaa31d2a10412e4be [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:25:57.900007 31590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e89196258634e11ae52fa1d0cd62c79" candidate_uuid: "2831154de99842959a694f57c980b448" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a16f2984931649a2a594bdd90b4debae"
I20260810 03:25:57.900218 31590 raft_consensus.cc:3037] T 6e89196258634e11ae52fa1d0cd62c79 P a16f2984931649a2a594bdd90b4debae [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:25:57.902544 31358 raft_consensus.cc:2454] T 6e89196258634e11ae52fa1d0cd62c79 P d9f56db87b6542bfaa31d2a10412e4be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2831154de99842959a694f57c980b448 in term 1.
I20260810 03:25:57.903028 31045 leader_election.cc:302] T 6e89196258634e11ae52fa1d0cd62c79 P 2831154de99842959a694f57c980b448 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2831154de99842959a694f57c980b448, d9f56db87b6542bfaa31d2a10412e4be; no voters:
I20260810 03:25:57.903250 31590 raft_consensus.cc:2454] T 6e89196258634e11ae52fa1d0cd62c79 P a16f2984931649a2a594bdd90b4debae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2831154de99842959a694f57c980b448 in term 1.
I20260810 03:25:57.903350 31848 raft_consensus.cc:2781] T 6e89196258634e11ae52fa1d0cd62c79 P 2831154de99842959a694f57c980b448 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:25:57.903865 31848 raft_consensus.cc:686] T 6e89196258634e11ae52fa1d0cd62c79 P 2831154de99842959a694f57c980b448 [term 1 LEADER]: Becoming Leader. State: Replica: 2831154de99842959a694f57c980b448, State: Running, Role: LEADER
I20260810 03:25:57.904263 31848 consensus_queue.cc:237] T 6e89196258634e11ae52fa1d0cd62c79 P 2831154de99842959a694f57c980b448 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2831154de99842959a694f57c980b448" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 33653 } } peers { permanent_uuid: "d9f56db87b6542bfaa31d2a10412e4be" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 33631 } } peers { permanent_uuid: "a16f2984931649a2a594bdd90b4debae" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 42247 } }
I20260810 03:25:57.908133 30908 catalog_manager.cc:5495] T 6e89196258634e11ae52fa1d0cd62c79 P 2831154de99842959a694f57c980b448 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2831154de99842959a694f57c980b448 (127.27.161.1). New cstate: current_term: 1 leader_uuid: "2831154de99842959a694f57c980b448" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2831154de99842959a694f57c980b448" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 33653 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d9f56db87b6542bfaa31d2a10412e4be" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 33631 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a16f2984931649a2a594bdd90b4debae" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 42247 } health_report { overall_health: UNKNOWN } } }
I20260810 03:25:57.942059 31863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:57.936066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59476 (local address 127.27.161.1:33653)
0810 03:25:57.936622 (+ 556us) server_negotiation.cc:206] Beginning negotiation
0810 03:25:57.936626 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:25:57.936645 (+ 19us) server_negotiation.cc:407] Connection header received
0810 03:25:57.936730 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:25:57.936734 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:25:57.936816 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:25:57.936962 (+ 146us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 03:25:57.937403 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:57.938249 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:57.939652 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:57.940083 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:57.940197 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:25:57.940665 (+ 468us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 03:25:57.941319 (+ 654us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 03:25:57.941451 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0810 03:25:57.941567 (+ 116us) server_negotiation.cc:299] Negotiation successful
0810 03:25:57.941657 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":434,"thread_start_us":276,"threads_started":1}
I20260810 03:25:57.958127 31856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:57.952714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.161.1:40665 (local address 127.27.161.3:42247)
0810 03:25:57.952923 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0810 03:25:57.952930 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:25:57.953425 (+ 495us) server_negotiation.cc:407] Connection header received
0810 03:25:57.953626 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:25:57.953630 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:25:57.953697 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:25:57.953773 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 03:25:57.954300 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:57.955001 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:57.956367 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:57.956797 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:57.956909 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:25:57.957366 (+ 457us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 03:25:57.957477 (+ 111us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 03:25:57.957550 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0810 03:25:57.957686 (+ 136us) server_negotiation.cc:299] Negotiation successful
0810 03:25:57.957771 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260810 03:25:57.958189 31870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:57.952483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.3:42247 (local address 127.27.161.1:40665)
0810 03:25:57.953319 (+ 836us) negotiation.cc:107] Waiting for socket to connect
0810 03:25:57.953334 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 03:25:57.953489 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:25:57.953802 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:25:57.953806 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:25:57.953818 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 03:25:57.954145 (+ 327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:25:57.954156 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:25:57.955143 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:25:57.955148 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:25:57.956225 (+ 1077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:25:57.956238 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:25:57.956976 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:25:57.956983 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:25:57.957127 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:25:57.957190 (+ 63us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 03:25:57.957574 (+ 384us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 03:25:57.957586 (+ 12us) client_negotiation.cc:769] Sending connection context
0810 03:25:57.957654 (+ 68us) client_negotiation.cc:240] Negotiation successful
0810 03:25:57.957799 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":693,"thread_start_us":471,"threads_started":1}
I20260810 03:25:57.966858 31358 raft_consensus.cc:1261] T 6e89196258634e11ae52fa1d0cd62c79 P d9f56db87b6542bfaa31d2a10412e4be [term 1 FOLLOWER]: Refusing update from remote peer 2831154de99842959a694f57c980b448: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:25:57.966894 31590 raft_consensus.cc:1261] T 6e89196258634e11ae52fa1d0cd62c79 P a16f2984931649a2a594bdd90b4debae [term 1 FOLLOWER]: Refusing update from remote peer 2831154de99842959a694f57c980b448: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:25:57.967581 31857 consensus_queue.cc:1035] T 6e89196258634e11ae52fa1d0cd62c79 P 2831154de99842959a694f57c980b448 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9f56db87b6542bfaa31d2a10412e4be" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 33631 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:25:57.968070 31848 consensus_queue.cc:1035] T 6e89196258634e11ae52fa1d0cd62c79 P 2831154de99842959a694f57c980b448 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a16f2984931649a2a594bdd90b4debae" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 42247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:25:58.008739 31863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:57.994823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.161.3:49333 (local address 127.27.161.1:33653)
0810 03:25:57.994986 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0810 03:25:57.994990 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:25:58.003129 (+ 8139us) server_negotiation.cc:407] Connection header received
0810 03:25:58.003263 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:25:58.003267 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:25:58.003344 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:25:58.003419 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 03:25:58.003826 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:58.004811 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:58.006879 (+ 2068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:58.007456 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:58.007587 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:25:58.007992 (+ 405us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 03:25:58.008107 (+ 115us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 03:25:58.008184 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0810 03:25:58.008323 (+ 139us) server_negotiation.cc:299] Negotiation successful
0810 03:25:58.008405 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260810 03:25:58.008774 31877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:57.994578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.1:33653 (local address 127.27.161.3:49333)
0810 03:25:58.002992 (+ 8414us) negotiation.cc:107] Waiting for socket to connect
0810 03:25:58.003010 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 03:25:58.003144 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:25:58.003442 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:25:58.003445 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:25:58.003455 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 03:25:58.003701 (+ 246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:25:58.003710 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:25:58.005125 (+ 1415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:25:58.005132 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:25:58.006706 (+ 1574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:25:58.006723 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:25:58.007645 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:25:58.007653 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:25:58.007782 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:25:58.007826 (+ 44us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 03:25:58.008201 (+ 375us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 03:25:58.008213 (+ 12us) client_negotiation.cc:769] Sending connection context
0810 03:25:58.008285 (+ 72us) client_negotiation.cc:240] Negotiation successful
0810 03:25:58.008428 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8219,"thread_start_us":8002,"threads_started":1}
I20260810 03:25:58.041834 30908 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55680:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 03:25:58.042225 30908 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55680:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 03:25:58.045711 30908 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0733cfa08dc743d8a587b321ff9eba88: Sending DeleteTablet for 3 replicas of tablet 6e89196258634e11ae52fa1d0cd62c79
I20260810 03:25:58.046756 31126 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e89196258634e11ae52fa1d0cd62c79 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:25:58 UTC) from {username='kudu', principal='kudu/127.27.161.62@KRBTEST.COM'} at 127.0.0.1:59472
I20260810 03:25:58.047075 31826 tablet_replica.cc:330] T 6e89196258634e11ae52fa1d0cd62c79 P 2831154de99842959a694f57c980b448: stopping tablet replica
I20260810 03:25:58.047127 31331 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e89196258634e11ae52fa1d0cd62c79 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:25:58 UTC) from {username='kudu', principal='kudu/127.27.161.62@KRBTEST.COM'} at 127.0.0.1:40112
I20260810 03:25:58.047281 31826 raft_consensus.cc:2229] T 6e89196258634e11ae52fa1d0cd62c79 P 2831154de99842959a694f57c980b448 [term 1 LEADER]: Raft consensus shutting down.
I20260810 03:25:58.047441 31829 tablet_replica.cc:330] T 6e89196258634e11ae52fa1d0cd62c79 P d9f56db87b6542bfaa31d2a10412e4be: stopping tablet replica
I20260810 03:25:58.047657 31829 raft_consensus.cc:2229] T 6e89196258634e11ae52fa1d0cd62c79 P d9f56db87b6542bfaa31d2a10412e4be [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 03:25:58.047873 31829 raft_consensus.cc:2258] T 6e89196258634e11ae52fa1d0cd62c79 P d9f56db87b6542bfaa31d2a10412e4be [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:25:58.047932 31826 raft_consensus.cc:2258] T 6e89196258634e11ae52fa1d0cd62c79 P 2831154de99842959a694f57c980b448 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:25:58.048282 31559 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e89196258634e11ae52fa1d0cd62c79 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:25:58 UTC) from {username='kudu', principal='kudu/127.27.161.62@KRBTEST.COM'} at 127.0.0.1:44192
I20260810 03:25:58.048585 31827 tablet_replica.cc:330] T 6e89196258634e11ae52fa1d0cd62c79 P a16f2984931649a2a594bdd90b4debae: stopping tablet replica
I20260810 03:25:58.048795 31827 raft_consensus.cc:2229] T 6e89196258634e11ae52fa1d0cd62c79 P a16f2984931649a2a594bdd90b4debae [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 03:25:58.048996 31827 raft_consensus.cc:2258] T 6e89196258634e11ae52fa1d0cd62c79 P a16f2984931649a2a594bdd90b4debae [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:25:58.049079 31829 ts_tablet_manager.cc:1905] T 6e89196258634e11ae52fa1d0cd62c79 P d9f56db87b6542bfaa31d2a10412e4be: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:25:58.050436 31827 ts_tablet_manager.cc:1905] T 6e89196258634e11ae52fa1d0cd62c79 P a16f2984931649a2a594bdd90b4debae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:25:58.055102 31826 ts_tablet_manager.cc:1905] T 6e89196258634e11ae52fa1d0cd62c79 P 2831154de99842959a694f57c980b448: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:25:58.056152 31829 ts_tablet_manager.cc:1918] T 6e89196258634e11ae52fa1d0cd62c79 P d9f56db87b6542bfaa31d2a10412e4be: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 03:25:58.056352 31829 log.cc:1196] T 6e89196258634e11ae52fa1d0cd62c79 P d9f56db87b6542bfaa31d2a10412e4be: Deleting WAL directory at /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786332336319544-28292-0/minicluster-data/ts-1/wal/wals/6e89196258634e11ae52fa1d0cd62c79
I20260810 03:25:58.056773 31829 ts_tablet_manager.cc:1939] T 6e89196258634e11ae52fa1d0cd62c79 P d9f56db87b6542bfaa31d2a10412e4be: Deleting consensus metadata
I20260810 03:25:58.058058 30888 catalog_manager.cc:4841] TS d9f56db87b6542bfaa31d2a10412e4be (127.27.161.2:33631): tablet 6e89196258634e11ae52fa1d0cd62c79 (table test-table [id=fc0fb98a4b9e464daa1ec0bdf4452948]) successfully deleted
I20260810 03:25:58.058816 30908 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55680:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 03:25:58.059562 30908 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 03:25:58.061865 31827 ts_tablet_manager.cc:1918] T 6e89196258634e11ae52fa1d0cd62c79 P a16f2984931649a2a594bdd90b4debae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 03:25:58.062062 31827 log.cc:1196] T 6e89196258634e11ae52fa1d0cd62c79 P a16f2984931649a2a594bdd90b4debae: Deleting WAL directory at /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786332336319544-28292-0/minicluster-data/ts-2/wal/wals/6e89196258634e11ae52fa1d0cd62c79
I20260810 03:25:58.062510 31827 ts_tablet_manager.cc:1939] T 6e89196258634e11ae52fa1d0cd62c79 P a16f2984931649a2a594bdd90b4debae: Deleting consensus metadata
I20260810 03:25:58.063654 30888 catalog_manager.cc:4841] TS a16f2984931649a2a594bdd90b4debae (127.27.161.3:42247): tablet 6e89196258634e11ae52fa1d0cd62c79 (table test-table [id=fc0fb98a4b9e464daa1ec0bdf4452948]) successfully deleted
I20260810 03:25:58.065367 31826 ts_tablet_manager.cc:1918] T 6e89196258634e11ae52fa1d0cd62c79 P 2831154de99842959a694f57c980b448: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 03:25:58.065551 31826 log.cc:1196] T 6e89196258634e11ae52fa1d0cd62c79 P 2831154de99842959a694f57c980b448: Deleting WAL directory at /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786332336319544-28292-0/minicluster-data/ts-0/wal/wals/6e89196258634e11ae52fa1d0cd62c79
I20260810 03:25:58.066140 31826 ts_tablet_manager.cc:1939] T 6e89196258634e11ae52fa1d0cd62c79 P 2831154de99842959a694f57c980b448: Deleting consensus metadata
I20260810 03:25:58.067518 30887 catalog_manager.cc:4841] TS 2831154de99842959a694f57c980b448 (127.27.161.1:33653): tablet 6e89196258634e11ae52fa1d0cd62c79 (table test-table [id=fc0fb98a4b9e464daa1ec0bdf4452948]) successfully deleted
I20260810 03:25:58.067487 31331 tablet_service.cc:1467] Processing CreateTablet for tablet 3d3cd936f14d4bcc99e031d6e57e5d4d (DEFAULT_TABLE table=test-table [id=577b02a7687a421291be8c8a0c868732]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:25:58.067916 31331 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d3cd936f14d4bcc99e031d6e57e5d4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:25:58.067875 31126 tablet_service.cc:1467] Processing CreateTablet for tablet 3d3cd936f14d4bcc99e031d6e57e5d4d (DEFAULT_TABLE table=test-table [id=577b02a7687a421291be8c8a0c868732]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:25:58.068282 31126 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d3cd936f14d4bcc99e031d6e57e5d4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:25:58.072703 31843 tablet_bootstrap.cc:492] T 3d3cd936f14d4bcc99e031d6e57e5d4d P d9f56db87b6542bfaa31d2a10412e4be: Bootstrap starting.
I20260810 03:25:58.074563 31843 tablet_bootstrap.cc:654] T 3d3cd936f14d4bcc99e031d6e57e5d4d P d9f56db87b6542bfaa31d2a10412e4be: Neither blocks nor log segments found. Creating new log.
I20260810 03:25:58.075196 31559 tablet_service.cc:1467] Processing CreateTablet for tablet 3d3cd936f14d4bcc99e031d6e57e5d4d (DEFAULT_TABLE table=test-table [id=577b02a7687a421291be8c8a0c868732]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:25:58.075625 31559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d3cd936f14d4bcc99e031d6e57e5d4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:25:58.075914 31840 tablet_bootstrap.cc:492] T 3d3cd936f14d4bcc99e031d6e57e5d4d P 2831154de99842959a694f57c980b448: Bootstrap starting.
I20260810 03:25:58.077852 31840 tablet_bootstrap.cc:654] T 3d3cd936f14d4bcc99e031d6e57e5d4d P 2831154de99842959a694f57c980b448: Neither blocks nor log segments found. Creating new log.
I20260810 03:25:58.082927 31842 tablet_bootstrap.cc:492] T 3d3cd936f14d4bcc99e031d6e57e5d4d P a16f2984931649a2a594bdd90b4debae: Bootstrap starting.
I20260810 03:25:58.084944 31842 tablet_bootstrap.cc:654] T 3d3cd936f14d4bcc99e031d6e57e5d4d P a16f2984931649a2a594bdd90b4debae: Neither blocks nor log segments found. Creating new log.
I20260810 03:25:58.093003 31840 tablet_bootstrap.cc:492] T 3d3cd936f14d4bcc99e031d6e57e5d4d P 2831154de99842959a694f57c980b448: No bootstrap required, opened a new log
I20260810 03:25:58.093246 31840 ts_tablet_manager.cc:1397] T 3d3cd936f14d4bcc99e031d6e57e5d4d P 2831154de99842959a694f57c980b448: Time spent bootstrapping tablet: real 0.017s user 0.007s sys 0.008s
I20260810 03:25:58.094403 31840 raft_consensus.cc:348] T 3d3cd936f14d4bcc99e031d6e57e5d4d P 2831154de99842959a694f57c980b448 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2831154de99842959a694f57c980b448" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 33653 } } peers { permanent_uuid: "d9f56db87b6542bfaa31d2a10412e4be" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 33631 } } peers { permanent_uuid: "a16f2984931649a2a594bdd90b4debae" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 42247 } }
I20260810 03:25:58.094712 31840 raft_consensus.cc:374] T 3d3cd936f14d4bcc99e031d6e57e5d4d P 2831154de99842959a694f57c980b448 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:25:58.094823 31840 raft_consensus.cc:729] T 3d3cd936f14d4bcc99e031d6e57e5d4d P 2831154de99842959a694f57c980b448 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2831154de99842959a694f57c980b448, State: Initialized, Role: FOLLOWER
I20260810 03:25:58.095134 31840 consensus_queue.cc:260] T 3d3cd936f14d4bcc99e031d6e57e5d4d P 2831154de99842959a694f57c980b448 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2831154de99842959a694f57c980b448" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 33653 } } peers { permanent_uuid: "d9f56db87b6542bfaa31d2a10412e4be" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 33631 } } peers { permanent_uuid: "a16f2984931649a2a594bdd90b4debae" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 42247 } }
I20260810 03:25:58.096724 31840 ts_tablet_manager.cc:1428] T 3d3cd936f14d4bcc99e031d6e57e5d4d P 2831154de99842959a694f57c980b448: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260810 03:25:58.098973 31842 tablet_bootstrap.cc:492] T 3d3cd936f14d4bcc99e031d6e57e5d4d P a16f2984931649a2a594bdd90b4debae: No bootstrap required, opened a new log
I20260810 03:25:58.099289 31843 tablet_bootstrap.cc:492] T 3d3cd936f14d4bcc99e031d6e57e5d4d P d9f56db87b6542bfaa31d2a10412e4be: No bootstrap required, opened a new log
I20260810 03:25:58.099485 31843 ts_tablet_manager.cc:1397] T 3d3cd936f14d4bcc99e031d6e57e5d4d P d9f56db87b6542bfaa31d2a10412e4be: Time spent bootstrapping tablet: real 0.027s user 0.006s sys 0.016s
I20260810 03:25:58.100378 31843 raft_consensus.cc:348] T 3d3cd936f14d4bcc99e031d6e57e5d4d P d9f56db87b6542bfaa31d2a10412e4be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2831154de99842959a694f57c980b448" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 33653 } } peers { permanent_uuid: "d9f56db87b6542bfaa31d2a10412e4be" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 33631 } } peers { permanent_uuid: "a16f2984931649a2a594bdd90b4debae" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 42247 } }
I20260810 03:25:58.100667 31843 raft_consensus.cc:374] T 3d3cd936f14d4bcc99e031d6e57e5d4d P d9f56db87b6542bfaa31d2a10412e4be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:25:58.100687 31842 ts_tablet_manager.cc:1397] T 3d3cd936f14d4bcc99e031d6e57e5d4d P a16f2984931649a2a594bdd90b4debae: Time spent bootstrapping tablet: real 0.018s user 0.006s sys 0.004s
I20260810 03:25:58.100764 31843 raft_consensus.cc:729] T 3d3cd936f14d4bcc99e031d6e57e5d4d P d9f56db87b6542bfaa31d2a10412e4be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9f56db87b6542bfaa31d2a10412e4be, State: Initialized, Role: FOLLOWER
I20260810 03:25:58.101027 31843 consensus_queue.cc:260] T 3d3cd936f14d4bcc99e031d6e57e5d4d P d9f56db87b6542bfaa31d2a10412e4be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2831154de99842959a694f57c980b448" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 33653 } } peers { permanent_uuid: "d9f56db87b6542bfaa31d2a10412e4be" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 33631 } } peers { permanent_uuid: "a16f2984931649a2a594bdd90b4debae" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 42247 } }
I20260810 03:25:58.101657 31843 ts_tablet_manager.cc:1428] T 3d3cd936f14d4bcc99e031d6e57e5d4d P d9f56db87b6542bfaa31d2a10412e4be: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260810 03:25:58.101647 31842 raft_consensus.cc:348] T 3d3cd936f14d4bcc99e031d6e57e5d4d P a16f2984931649a2a594bdd90b4debae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2831154de99842959a694f57c980b448" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 33653 } } peers { permanent_uuid: "d9f56db87b6542bfaa31d2a10412e4be" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 33631 } } peers { permanent_uuid: "a16f2984931649a2a594bdd90b4debae" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 42247 } }
I20260810 03:25:58.102202 31842 raft_consensus.cc:374] T 3d3cd936f14d4bcc99e031d6e57e5d4d P a16f2984931649a2a594bdd90b4debae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:25:58.102301 31842 raft_consensus.cc:729] T 3d3cd936f14d4bcc99e031d6e57e5d4d P a16f2984931649a2a594bdd90b4debae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a16f2984931649a2a594bdd90b4debae, State: Initialized, Role: FOLLOWER
I20260810 03:25:58.102567 31842 consensus_queue.cc:260] T 3d3cd936f14d4bcc99e031d6e57e5d4d P a16f2984931649a2a594bdd90b4debae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2831154de99842959a694f57c980b448" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 33653 } } peers { permanent_uuid: "d9f56db87b6542bfaa31d2a10412e4be" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 33631 } } peers { permanent_uuid: "a16f2984931649a2a594bdd90b4debae" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 42247 } }
I20260810 03:25:58.103137 31842 ts_tablet_manager.cc:1428] T 3d3cd936f14d4bcc99e031d6e57e5d4d P a16f2984931649a2a594bdd90b4debae: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260810 03:25:58.120240 31796 raft_consensus.cc:482] T 3d3cd936f14d4bcc99e031d6e57e5d4d P a16f2984931649a2a594bdd90b4debae [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 03:25:58.120501 31796 raft_consensus.cc:504] T 3d3cd936f14d4bcc99e031d6e57e5d4d P a16f2984931649a2a594bdd90b4debae [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2831154de99842959a694f57c980b448" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 33653 } } peers { permanent_uuid: "d9f56db87b6542bfaa31d2a10412e4be" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 33631 } } peers { permanent_uuid: "a16f2984931649a2a594bdd90b4debae" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 42247 } }
I20260810 03:25:58.121356 31796 leader_election.cc:288] T 3d3cd936f14d4bcc99e031d6e57e5d4d P a16f2984931649a2a594bdd90b4debae [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2831154de99842959a694f57c980b448 (127.27.161.1:33653), d9f56db87b6542bfaa31d2a10412e4be (127.27.161.2:33631)
I20260810 03:25:58.122045 31151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d3cd936f14d4bcc99e031d6e57e5d4d" candidate_uuid: "a16f2984931649a2a594bdd90b4debae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2831154de99842959a694f57c980b448" is_pre_election: true
I20260810 03:25:58.122041 31358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d3cd936f14d4bcc99e031d6e57e5d4d" candidate_uuid: "a16f2984931649a2a594bdd90b4debae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9f56db87b6542bfaa31d2a10412e4be" is_pre_election: true
I20260810 03:25:58.122295 31358 raft_consensus.cc:2454] T 3d3cd936f14d4bcc99e031d6e57e5d4d P d9f56db87b6542bfaa31d2a10412e4be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a16f2984931649a2a594bdd90b4debae in term 0.
I20260810 03:25:58.122326 31151 raft_consensus.cc:2454] T 3d3cd936f14d4bcc99e031d6e57e5d4d P 2831154de99842959a694f57c980b448 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a16f2984931649a2a594bdd90b4debae in term 0.
I20260810 03:25:58.122762 31481 leader_election.cc:302] T 3d3cd936f14d4bcc99e031d6e57e5d4d P a16f2984931649a2a594bdd90b4debae [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a16f2984931649a2a594bdd90b4debae, d9f56db87b6542bfaa31d2a10412e4be; no voters:
I20260810 03:25:58.123099 31796 raft_consensus.cc:2781] T 3d3cd936f14d4bcc99e031d6e57e5d4d P a16f2984931649a2a594bdd90b4debae [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 03:25:58.123239 31796 raft_consensus.cc:482] T 3d3cd936f14d4bcc99e031d6e57e5d4d P a16f2984931649a2a594bdd90b4debae [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 03:25:58.123342 31796 raft_consensus.cc:3037] T 3d3cd936f14d4bcc99e031d6e57e5d4d P a16f2984931649a2a594bdd90b4debae [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:25:58.125711 31796 raft_consensus.cc:504] T 3d3cd936f14d4bcc99e031d6e57e5d4d P a16f2984931649a2a594bdd90b4debae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2831154de99842959a694f57c980b448" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 33653 } } peers { permanent_uuid: "d9f56db87b6542bfaa31d2a10412e4be" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 33631 } } peers { permanent_uuid: "a16f2984931649a2a594bdd90b4debae" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 42247 } }
I20260810 03:25:58.126343 31796 leader_election.cc:288] T 3d3cd936f14d4bcc99e031d6e57e5d4d P a16f2984931649a2a594bdd90b4debae [CANDIDATE]: Term 1 election: Requested vote from peers 2831154de99842959a694f57c980b448 (127.27.161.1:33653), d9f56db87b6542bfaa31d2a10412e4be (127.27.161.2:33631)
I20260810 03:25:58.126960 31358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d3cd936f14d4bcc99e031d6e57e5d4d" candidate_uuid: "a16f2984931649a2a594bdd90b4debae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9f56db87b6542bfaa31d2a10412e4be"
I20260810 03:25:58.127063 31151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d3cd936f14d4bcc99e031d6e57e5d4d" candidate_uuid: "a16f2984931649a2a594bdd90b4debae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2831154de99842959a694f57c980b448"
I20260810 03:25:58.127193 31358 raft_consensus.cc:3037] T 3d3cd936f14d4bcc99e031d6e57e5d4d P d9f56db87b6542bfaa31d2a10412e4be [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:25:58.127312 31151 raft_consensus.cc:3037] T 3d3cd936f14d4bcc99e031d6e57e5d4d P 2831154de99842959a694f57c980b448 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:25:58.129590 31358 raft_consensus.cc:2454] T 3d3cd936f14d4bcc99e031d6e57e5d4d P d9f56db87b6542bfaa31d2a10412e4be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a16f2984931649a2a594bdd90b4debae in term 1.
I20260810 03:25:58.130311 31481 leader_election.cc:302] T 3d3cd936f14d4bcc99e031d6e57e5d4d P a16f2984931649a2a594bdd90b4debae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a16f2984931649a2a594bdd90b4debae, d9f56db87b6542bfaa31d2a10412e4be; no voters:
I20260810 03:25:58.130856 31151 raft_consensus.cc:2454] T 3d3cd936f14d4bcc99e031d6e57e5d4d P 2831154de99842959a694f57c980b448 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a16f2984931649a2a594bdd90b4debae in term 1.
I20260810 03:25:58.131637 31796 raft_consensus.cc:2781] T 3d3cd936f14d4bcc99e031d6e57e5d4d P a16f2984931649a2a594bdd90b4debae [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:25:58.131990 31796 raft_consensus.cc:686] T 3d3cd936f14d4bcc99e031d6e57e5d4d P a16f2984931649a2a594bdd90b4debae [term 1 LEADER]: Becoming Leader. State: Replica: a16f2984931649a2a594bdd90b4debae, State: Running, Role: LEADER
I20260810 03:25:58.132437 31796 consensus_queue.cc:237] T 3d3cd936f14d4bcc99e031d6e57e5d4d P a16f2984931649a2a594bdd90b4debae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2831154de99842959a694f57c980b448" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 33653 } } peers { permanent_uuid: "d9f56db87b6542bfaa31d2a10412e4be" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 33631 } } peers { permanent_uuid: "a16f2984931649a2a594bdd90b4debae" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 42247 } }
I20260810 03:25:58.136032 30906 catalog_manager.cc:5495] T 3d3cd936f14d4bcc99e031d6e57e5d4d P a16f2984931649a2a594bdd90b4debae reported cstate change: term changed from 0 to 1, leader changed from <none> to a16f2984931649a2a594bdd90b4debae (127.27.161.3). New cstate: current_term: 1 leader_uuid: "a16f2984931649a2a594bdd90b4debae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2831154de99842959a694f57c980b448" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 33653 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d9f56db87b6542bfaa31d2a10412e4be" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 33631 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a16f2984931649a2a594bdd90b4debae" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 42247 } health_report { overall_health: HEALTHY } } }
I20260810 03:25:58.160064 31856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:58.154675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44220 (local address 127.27.161.3:42247)
0810 03:25:58.154861 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0810 03:25:58.154867 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:25:58.155129 (+ 262us) server_negotiation.cc:407] Connection header received
0810 03:25:58.155319 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:25:58.155325 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:25:58.155398 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:25:58.155472 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 03:25:58.155867 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:58.156815 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:58.158171 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:58.158747 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:58.158868 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:25:58.159354 (+ 486us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 03:25:58.159467 (+ 113us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 03:25:58.159541 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0810 03:25:58.159694 (+ 153us) server_negotiation.cc:299] Negotiation successful
0810 03:25:58.159789 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260810 03:25:58.170046 31358 raft_consensus.cc:1261] T 3d3cd936f14d4bcc99e031d6e57e5d4d P d9f56db87b6542bfaa31d2a10412e4be [term 1 FOLLOWER]: Refusing update from remote peer a16f2984931649a2a594bdd90b4debae: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:25:58.170928 31151 raft_consensus.cc:1261] T 3d3cd936f14d4bcc99e031d6e57e5d4d P 2831154de99842959a694f57c980b448 [term 1 FOLLOWER]: Refusing update from remote peer a16f2984931649a2a594bdd90b4debae: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:25:58.171293 31806 consensus_queue.cc:1035] T 3d3cd936f14d4bcc99e031d6e57e5d4d P a16f2984931649a2a594bdd90b4debae [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9f56db87b6542bfaa31d2a10412e4be" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 33631 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:25:58.171881 31796 consensus_queue.cc:1035] T 3d3cd936f14d4bcc99e031d6e57e5d4d P a16f2984931649a2a594bdd90b4debae [LEADER]: Connected to new peer: Peer: permanent_uuid: "2831154de99842959a694f57c980b448" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 33653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260810 03:25:58.197726 31797 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260810 03:25:59.820587 30906 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55680:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 03:25:59.820869 30906 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55680:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 03:25:59.829252 30906 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0733cfa08dc743d8a587b321ff9eba88: Sending DeleteTablet for 3 replicas of tablet 3d3cd936f14d4bcc99e031d6e57e5d4d
I20260810 03:25:59.830346 31331 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d3cd936f14d4bcc99e031d6e57e5d4d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:25:59 UTC) from {username='kudu', principal='kudu/127.27.161.62@KRBTEST.COM'} at 127.0.0.1:40112
I20260810 03:25:59.830403 31126 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d3cd936f14d4bcc99e031d6e57e5d4d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:25:59 UTC) from {username='kudu', principal='kudu/127.27.161.62@KRBTEST.COM'} at 127.0.0.1:59472
I20260810 03:25:59.830541 31559 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d3cd936f14d4bcc99e031d6e57e5d4d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:25:59 UTC) from {username='kudu', principal='kudu/127.27.161.62@KRBTEST.COM'} at 127.0.0.1:44192
I20260810 03:25:59.831007 28292 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iBH5/build/asan/bin/kudu pid 31020
I20260810 03:25:59.843519 31928 tablet_replica.cc:330] T 3d3cd936f14d4bcc99e031d6e57e5d4d P d9f56db87b6542bfaa31d2a10412e4be: stopping tablet replica
I20260810 03:25:59.843747 31930 tablet_replica.cc:330] T 3d3cd936f14d4bcc99e031d6e57e5d4d P 2831154de99842959a694f57c980b448: stopping tablet replica
I20260810 03:25:59.843817 31931 tablet_replica.cc:330] T 3d3cd936f14d4bcc99e031d6e57e5d4d P a16f2984931649a2a594bdd90b4debae: stopping tablet replica
I20260810 03:25:59.844197 31930 raft_consensus.cc:2229] T 3d3cd936f14d4bcc99e031d6e57e5d4d P 2831154de99842959a694f57c980b448 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 03:25:59.844261 31928 raft_consensus.cc:2229] T 3d3cd936f14d4bcc99e031d6e57e5d4d P d9f56db87b6542bfaa31d2a10412e4be [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 03:25:59.844345 31931 raft_consensus.cc:2229] T 3d3cd936f14d4bcc99e031d6e57e5d4d P a16f2984931649a2a594bdd90b4debae [term 1 LEADER]: Raft consensus shutting down.
I20260810 03:25:59.844732 31930 raft_consensus.cc:2258] T 3d3cd936f14d4bcc99e031d6e57e5d4d P 2831154de99842959a694f57c980b448 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:25:59.844769 31928 raft_consensus.cc:2258] T 3d3cd936f14d4bcc99e031d6e57e5d4d P d9f56db87b6542bfaa31d2a10412e4be [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:25:59.845367 31931 raft_consensus.cc:2258] T 3d3cd936f14d4bcc99e031d6e57e5d4d P a16f2984931649a2a594bdd90b4debae [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:25:59.845933 31930 ts_tablet_manager.cc:1905] T 3d3cd936f14d4bcc99e031d6e57e5d4d P 2831154de99842959a694f57c980b448: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:25:59.845937 31928 ts_tablet_manager.cc:1905] T 3d3cd936f14d4bcc99e031d6e57e5d4d P d9f56db87b6542bfaa31d2a10412e4be: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:25:59.847075 31931 ts_tablet_manager.cc:1905] T 3d3cd936f14d4bcc99e031d6e57e5d4d P a16f2984931649a2a594bdd90b4debae: Deleting tablet data with delete state TABLET_DATA_DELETED
Aug 10 03:25:59 dist-test-slave-vpvm krb5kdc[30840](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332357, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.27.161.1@KRBTEST.COM
I20260810 03:25:59.867700 31933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:59.832825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42512 (local address 127.27.161.1:33653)
0810 03:25:59.847069 (+ 14244us) server_negotiation.cc:206] Beginning negotiation
0810 03:25:59.847075 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:25:59.847102 (+ 27us) server_negotiation.cc:407] Connection header received
0810 03:25:59.847180 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:25:59.847185 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:25:59.847289 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:25:59.847410 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:25:59.849377 (+ 1967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:59.850557 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:59.851964 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:59.852422 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:59.852559 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:25:59.861027 (+ 8468us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:25:59.861049 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:25:59.861068 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:25:59.861110 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:25:59.864583 (+ 3473us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:25:59.865396 (+ 813us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:25:59.865405 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:25:59.865408 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:25:59.865692 (+ 284us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:25:59.866162 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:25:59.866166 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:25:59.866168 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:25:59.866571 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:25:59.866845 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0810 03:25:59.867181 (+ 336us) server_negotiation.cc:299] Negotiation successful
0810 03:25:59.867403 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13072,"thread_start_us":13487,"threads_started":1}
W20260810 03:25:59.868429 31217 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42512
W20260810 03:25:59.869040 28292 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task24iBH5/build/asan/bin/kudu pid 31020: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260810 03:25:59.869262 28292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iBH5/build/asan/bin/kudu with pid 31020
I20260810 03:25:59.928944 31928 ts_tablet_manager.cc:1918] T 3d3cd936f14d4bcc99e031d6e57e5d4d P d9f56db87b6542bfaa31d2a10412e4be: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260810 03:25:59.928951 31931 ts_tablet_manager.cc:1918] T 3d3cd936f14d4bcc99e031d6e57e5d4d P a16f2984931649a2a594bdd90b4debae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260810 03:25:59.929147 31931 log.cc:1196] T 3d3cd936f14d4bcc99e031d6e57e5d4d P a16f2984931649a2a594bdd90b4debae: Deleting WAL directory at /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786332336319544-28292-0/minicluster-data/ts-2/wal/wals/3d3cd936f14d4bcc99e031d6e57e5d4d
I20260810 03:25:59.929271 31928 log.cc:1196] T 3d3cd936f14d4bcc99e031d6e57e5d4d P d9f56db87b6542bfaa31d2a10412e4be: Deleting WAL directory at /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786332336319544-28292-0/minicluster-data/ts-1/wal/wals/3d3cd936f14d4bcc99e031d6e57e5d4d
I20260810 03:25:59.929814 31928 ts_tablet_manager.cc:1939] T 3d3cd936f14d4bcc99e031d6e57e5d4d P d9f56db87b6542bfaa31d2a10412e4be: Deleting consensus metadata
I20260810 03:25:59.929822 31931 ts_tablet_manager.cc:1939] T 3d3cd936f14d4bcc99e031d6e57e5d4d P a16f2984931649a2a594bdd90b4debae: Deleting consensus metadata
I20260810 03:25:59.952225 28292 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iBH5/build/asan/bin/kudu pid 31231
W20260810 03:25:59.952126 30887 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.161.1:33653 (error 108)
I20260810 03:25:59.954738 31938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:59.953854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.1:33653 (local address 127.0.0.1:42528)
0810 03:25:59.954369 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0810 03:25:59.954523 (+ 154us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.1:33653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":275,"threads_started":1}
W20260810 03:25:59.954985 30887 catalog_manager.cc:4584] TS 2831154de99842959a694f57c980b448 (127.27.161.1:33653): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3d3cd936f14d4bcc99e031d6e57e5d4d: Network error: Client connection negotiation failed: client connection to 127.27.161.1:33653: connect: Connection refused (error 111)
Aug 10 03:25:59 dist-test-slave-vpvm krb5kdc[30840](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332357, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.27.161.2@KRBTEST.COM
I20260810 03:25:59.970104 31935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:59.953049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50822 (local address 127.27.161.2:33631)
0810 03:25:59.954120 (+ 1071us) server_negotiation.cc:206] Beginning negotiation
0810 03:25:59.954125 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:25:59.954144 (+ 19us) server_negotiation.cc:407] Connection header received
0810 03:25:59.954202 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:25:59.954205 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:25:59.954271 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:25:59.955012 (+ 741us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:25:59.956461 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:59.957262 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:59.958650 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:59.959060 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:59.959225 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:25:59.963654 (+ 4429us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:25:59.963673 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:25:59.963691 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:25:59.963739 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:25:59.966768 (+ 3029us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:25:59.967683 (+ 915us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:25:59.967692 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:25:59.967694 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:25:59.967887 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:25:59.968241 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:25:59.968246 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:25:59.968248 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:25:59.968999 (+ 751us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:25:59.969299 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0810 03:25:59.969566 (+ 267us) server_negotiation.cc:299] Negotiation successful
0810 03:25:59.969771 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":833,"thread_start_us":722,"threads_started":1}
W20260810 03:25:59.970818 31439 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50822
W20260810 03:25:59.971660 28292 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task24iBH5/build/asan/bin/kudu pid 31231: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260810 03:25:59.971838 28292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iBH5/build/asan/bin/kudu with pid 31231
I20260810 03:25:59.979215 30888 catalog_manager.cc:4841] TS a16f2984931649a2a594bdd90b4debae (127.27.161.3:42247): tablet 3d3cd936f14d4bcc99e031d6e57e5d4d (table test-table [id=577b02a7687a421291be8c8a0c868732]) successfully deleted
I20260810 03:26:00.009390 31938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:00.008448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.1:33653 (local address 127.0.0.1:42532)
0810 03:26:00.008719 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0810 03:26:00.008835 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.1:33653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
I20260810 03:26:00.009552 28292 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iBH5/build/asan/bin/kudu pid 31455
I20260810 03:26:00.010330 31938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:00.009855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.2:33631 (local address 127.0.0.1:50824)
0810 03:26:00.010033 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0810 03:26:00.010147 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.2:33631: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260810 03:26:00.011018 31938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:00.010172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.1:33653 (local address 127.0.0.1:42548)
0810 03:26:00.010828 (+ 656us) negotiation.cc:107] Waiting for socket to connect
0810 03:26:00.010909 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.1:33653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
Aug 10 03:26:00 dist-test-slave-vpvm krb5kdc[30840](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332357, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.27.161.3@KRBTEST.COM
I20260810 03:26:00.027629 31940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:00.010313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51880 (local address 127.27.161.3:42247)
0810 03:26:00.010867 (+ 554us) server_negotiation.cc:206] Beginning negotiation
0810 03:26:00.010873 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:26:00.010890 (+ 17us) server_negotiation.cc:407] Connection header received
0810 03:26:00.010940 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:26:00.010944 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:26:00.011008 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:26:00.011133 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:26:00.012742 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:00.013470 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:00.014912 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:00.015310 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:00.015402 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:00.021266 (+ 5864us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:26:00.021285 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:26:00.021304 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:26:00.021348 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:26:00.024442 (+ 3094us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:26:00.025323 (+ 881us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:26:00.025330 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:26:00.025333 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:26:00.025477 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:26:00.025897 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:26:00.025901 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:26:00.025903 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:26:00.026408 (+ 505us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:26:00.026648 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0810 03:26:00.026996 (+ 348us) server_negotiation.cc:299] Negotiation successful
0810 03:26:00.027263 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":429,"thread_start_us":271,"threads_started":1}
W20260810 03:26:00.028354 31655 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51880
W20260810 03:26:00.029424 28292 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task24iBH5/build/asan/bin/kudu pid 31455: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260810 03:26:00.029626 28292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iBH5/build/asan/bin/kudu with pid 31455
I20260810 03:26:00.054913 31938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:00.054413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.2:33631 (local address 127.0.0.1:50836)
0810 03:26:00.054648 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0810 03:26:00.054741 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.2:33631: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110}
I20260810 03:26:00.056083 31938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:00.055718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.2:33631 (local address 127.0.0.1:50840)
0810 03:26:00.055892 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0810 03:26:00.055967 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.2:33631: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260810 03:26:00.059140 31938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:00.058753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.1:33653 (local address 127.0.0.1:42560)
0810 03:26:00.058915 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0810 03:26:00.058994 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.1:33653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260810 03:26:00.060113 31938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:00.059803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.1:33653 (local address 127.0.0.1:42562)
0810 03:26:00.059930 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0810 03:26:00.060010 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.1:33653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260810 03:26:00.070451 28292 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iBH5/build/asan/bin/kudu pid 30859
I20260810 03:26:00.071801 30973 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:26:00.196959 28292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iBH5/build/asan/bin/kudu with pid 30859
2026-08-10T03:26:00Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (8973 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786332336319544-28292-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 03:26:01 dist-test-slave-vpvm krb5kdc[31947](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 03:26:01 dist-test-slave-vpvm krb5kdc[31947](info): set up 2 sockets
Aug 10 03:26:01 dist-test-slave-vpvm krb5kdc[31947](info): commencing operation
krb5kdc: starting...
W20260810 03:26:02.217597 28292 mini_kdc.cc:121] Time spent starting KDC: real 1.896s user 0.004s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786332336319544-28292-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786332336319544-28292-0/krb5kdc/test-user.keytab.
Aug 10 03:26:02 dist-test-slave-vpvm krb5kdc[31947](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332362, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T03:26:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T03:26:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.161.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.161.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.161.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.62.keytab.
Entry for principal kudu/127.27.161.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.62.keytab.
I20260810 03:26:02.467242 28292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iBH5/build/asan/bin/kudu
/tmp/dist-test-task24iBH5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786332336319544-28292-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786332336319544-28292-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786332336319544-28292-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786332336319544-28292-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.161.62:33831
--webserver_interface=127.27.161.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.62.keytab
--principal=kudu/127.27.161.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.161.20:43587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.161.62:33831
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786332336319544-28292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786332336319544-28292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786332336319544-28292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:26:02.820689 31961 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:26:02.821004 31961 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:26:02.821142 31961 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:26:02.828619 31961 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:26:02.828749 31961 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:26:02.828806 31961 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:26:02.828886 31961 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:26:02.828951 31961 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:26:02.839296 31961 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.161.20:43587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786332336319544-28292-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786332336319544-28292-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.161.62:33831
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.62.keytab
--principal=kudu/127.27.161.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.161.62:33831
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786332336319544-28292-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.161.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786332336319544-28292-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:26:02.840771 31961 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:26:02.842263 31961 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 03:26:02.855443 31961 server_base.cc:1034] running on GCE node
W20260810 03:26:02.855412 31968 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:26:02.855383 31966 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:26:02.855463 31971 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:26:02.856937 31961 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:26:02.858551 31961 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:26:02.859774 31961 hybrid_clock.cc:648] HybridClock initialized: now 1786332362859723 us; error 36 us; skew 500 ppm
I20260810 03:26:02.860140 31961 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:26:02 dist-test-slave-vpvm krb5kdc[31947](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332362, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.161.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:26:02.863330 31961 init.cc:377] Logged in from keytab as kudu/127.27.161.62@KRBTEST.COM (short username kudu)
I20260810 03:26:02.865021 31961 webserver.cc:460] Webserver started at http://127.27.161.62:40549/ using document root <none> and password file <none>
I20260810 03:26:02.865756 31961 fs_manager.cc:359] Metadata directory not provided
I20260810 03:26:02.865878 31961 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:26:02.866230 31961 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:26:02.869383 31961 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786332336319544-28292-0/minicluster-data/master-0/data/instance:
uuid: "7e9fba918a8449ccbf3e083ef7769a52"
format_stamp: "Formatted at 2026-08-10 03:26:02 on dist-test-slave-vpvm"
server_key: "4d886bffd7638ee09b6854733d9ae1bd"
server_key_iv: "2cc79888a5827601fd3d18181e01caf5"
server_key_version: "encryptionkey@0"
I20260810 03:26:02.870312 31961 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786332336319544-28292-0/minicluster-data/master-0/wal/instance:
uuid: "7e9fba918a8449ccbf3e083ef7769a52"
format_stamp: "Formatted at 2026-08-10 03:26:02 on dist-test-slave-vpvm"
server_key: "4d886bffd7638ee09b6854733d9ae1bd"
server_key_iv: "2cc79888a5827601fd3d18181e01caf5"
server_key_version: "encryptionkey@0"
I20260810 03:26:02.875948 31961 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260810 03:26:02.880064 31988 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:26:02.881757 31961 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 03:26:02.882023 31961 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786332336319544-28292-0/minicluster-data/master-0/data,/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786332336319544-28292-0/minicluster-data/master-0/wal
uuid: "7e9fba918a8449ccbf3e083ef7769a52"
format_stamp: "Formatted at 2026-08-10 03:26:02 on dist-test-slave-vpvm"
server_key: "4d886bffd7638ee09b6854733d9ae1bd"
server_key_iv: "2cc79888a5827601fd3d18181e01caf5"
server_key_version: "encryptionkey@0"
I20260810 03:26:02.882272 31961 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786332336319544-28292-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786332336319544-28292-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786332336319544-28292-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:26:02.900275 31961 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:26:02.905112 31961 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:26:02.905421 31961 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:26:02.924242 31961 rpc_server.cc:307] RPC server started. Bound to: 127.27.161.62:33831
I20260810 03:26:02.924261 32066 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.161.62:33831 every 8 connection(s)
I20260810 03:26:02.925702 31961 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786332336319544-28292-0/minicluster-data/master-0/data/info.pb
I20260810 03:26:02.929576 32068 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:26:02.931993 28292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iBH5/build/asan/bin/kudu as pid 31961
I20260810 03:26:02.932245 28292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786332336319544-28292-0/minicluster-data/master-0/wal/instance
I20260810 03:26:02.933023 28292 external_mini_cluster.cc:1428] Setting key 67a241d5fd49a4cab1427e5917b0cb97
I20260810 03:26:02.942797 32068 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e9fba918a8449ccbf3e083ef7769a52: Bootstrap starting.
I20260810 03:26:02.946910 32068 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7e9fba918a8449ccbf3e083ef7769a52: Neither blocks nor log segments found. Creating new log.
I20260810 03:26:02.948215 32068 log.cc:824] T 00000000000000000000000000000000 P 7e9fba918a8449ccbf3e083ef7769a52: Log is configured to *not* fsync() on all Append() calls
Aug 10 03:26:02 dist-test-slave-vpvm krb5kdc[31947](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332362, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.161.62@KRBTEST.COM
I20260810 03:26:02.951967 32068 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e9fba918a8449ccbf3e083ef7769a52: No bootstrap required, opened a new log
I20260810 03:26:02.958606 32074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:02.935389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35030 (local address 127.27.161.62:33831)
0810 03:26:02.936273 (+ 884us) server_negotiation.cc:206] Beginning negotiation
0810 03:26:02.936285 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 03:26:02.936330 (+ 45us) server_negotiation.cc:407] Connection header received
0810 03:26:02.938123 (+ 1793us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:26:02.938149 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:26:02.938550 (+ 401us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:26:02.938900 (+ 350us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:26:02.940687 (+ 1787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:02.942055 (+ 1368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:02.943816 (+ 1761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:02.944517 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:02.944682 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:02.949534 (+ 4852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:26:02.949544 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:26:02.949560 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:26:02.949613 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:26:02.953715 (+ 4102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:26:02.954505 (+ 790us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:26:02.954516 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:26:02.954530 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:26:02.954775 (+ 245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:26:02.955175 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:26:02.955181 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:26:02.955184 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:26:02.955970 (+ 786us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:26:02.956286 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0810 03:26:02.956667 (+ 381us) server_negotiation.cc:299] Negotiation successful
0810 03:26:02.957127 (+ 460us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":658,"thread_start_us":377,"threads_started":1}
I20260810 03:26:02.961700 32068 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7e9fba918a8449ccbf3e083ef7769a52 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e9fba918a8449ccbf3e083ef7769a52" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 33831 } }
I20260810 03:26:02.962143 32068 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7e9fba918a8449ccbf3e083ef7769a52 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:26:02.962226 32068 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7e9fba918a8449ccbf3e083ef7769a52 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e9fba918a8449ccbf3e083ef7769a52, State: Initialized, Role: FOLLOWER
I20260810 03:26:02.962782 32068 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7e9fba918a8449ccbf3e083ef7769a52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e9fba918a8449ccbf3e083ef7769a52" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 33831 } }
I20260810 03:26:02.963003 32068 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7e9fba918a8449ccbf3e083ef7769a52 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 03:26:02.963104 32068 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7e9fba918a8449ccbf3e083ef7769a52 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 03:26:02.963249 32068 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7e9fba918a8449ccbf3e083ef7769a52 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:26:02.965474 32068 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7e9fba918a8449ccbf3e083ef7769a52 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e9fba918a8449ccbf3e083ef7769a52" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 33831 } }
I20260810 03:26:02.966028 32068 leader_election.cc:302] T 00000000000000000000000000000000 P 7e9fba918a8449ccbf3e083ef7769a52 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7e9fba918a8449ccbf3e083ef7769a52; no voters:
I20260810 03:26:02.966616 32068 leader_election.cc:288] T 00000000000000000000000000000000 P 7e9fba918a8449ccbf3e083ef7769a52 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 03:26:02.966774 32076 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7e9fba918a8449ccbf3e083ef7769a52 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:26:02.967770 32076 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7e9fba918a8449ccbf3e083ef7769a52 [term 1 LEADER]: Becoming Leader. State: Replica: 7e9fba918a8449ccbf3e083ef7769a52, State: Running, Role: LEADER
I20260810 03:26:02.968441 32076 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7e9fba918a8449ccbf3e083ef7769a52 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e9fba918a8449ccbf3e083ef7769a52" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 33831 } }
I20260810 03:26:02.968905 32068 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7e9fba918a8449ccbf3e083ef7769a52 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:26:02.971508 32077 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e9fba918a8449ccbf3e083ef7769a52 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7e9fba918a8449ccbf3e083ef7769a52" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e9fba918a8449ccbf3e083ef7769a52" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 33831 } } }
I20260810 03:26:02.971907 32077 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e9fba918a8449ccbf3e083ef7769a52 [sys.catalog]: This master's current role is: LEADER
I20260810 03:26:02.971297 32080 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e9fba918a8449ccbf3e083ef7769a52 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7e9fba918a8449ccbf3e083ef7769a52. Latest consensus state: current_term: 1 leader_uuid: "7e9fba918a8449ccbf3e083ef7769a52" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e9fba918a8449ccbf3e083ef7769a52" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 33831 } } }
I20260810 03:26:02.972520 32080 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e9fba918a8449ccbf3e083ef7769a52 [sys.catalog]: This master's current role is: LEADER
I20260810 03:26:02.974828 32093 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 03:26:02.979705 32093 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 03:26:02.989419 32093 catalog_manager.cc:1347] Generated new cluster ID: babea1f3687a46bfbd512b7763df1821
I20260810 03:26:02.989564 32093 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 03:26:03.000109 32093 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 03:26:03.002283 32093 catalog_manager.cc:1504] Loading token signing keys...
I20260810 03:26:03.018698 32093 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7e9fba918a8449ccbf3e083ef7769a52: Generated new TSK 0
I20260810 03:26:03.019996 32093 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.161.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.161.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.161.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.1.keytab.
Entry for principal kudu/127.27.161.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.1.keytab.
I20260810 03:26:03.084208 28292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iBH5/build/asan/bin/kudu
/tmp/dist-test-task24iBH5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786332336319544-28292-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786332336319544-28292-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786332336319544-28292-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786332336319544-28292-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.161.1:0
--local_ip_for_outbound_sockets=127.27.161.1
--webserver_interface=127.27.161.1
--webserver_port=0
--tserver_master_addrs=127.27.161.62:33831
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.1.keytab
--principal=kudu/127.27.161.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.161.20:43587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786332336319544-28292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786332336319544-28292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786332336319544-28292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:26:03.443956 32114 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:26:03.444315 32114 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:26:03.444478 32114 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:26:03.452715 32114 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:26:03.452893 32114 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:26:03.453068 32114 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.161.1
I20260810 03:26:03.463924 32114 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.161.20:43587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786332336319544-28292-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786332336319544-28292-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.1.keytab
--principal=kudu/127.27.161.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.161.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786332336319544-28292-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.161.1
--webserver_port=0
--tserver_master_addrs=127.27.161.62:33831
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.161.1
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786332336319544-28292-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:26:03.465407 32114 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:26:03.467054 32114 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:26:03.482201 32120 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:26:03.482203 32125 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:26:03.482228 32119 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:26:03.483096 32114 server_base.cc:1034] running on GCE node
I20260810 03:26:03.483845 32114 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:26:03.484879 32114 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:26:03.486054 32114 hybrid_clock.cc:648] HybridClock initialized: now 1786332363485989 us; error 53 us; skew 500 ppm
I20260810 03:26:03.486418 32114 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:26:03 dist-test-slave-vpvm krb5kdc[31947](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332363, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.161.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:26:03.490593 32114 init.cc:377] Logged in from keytab as kudu/127.27.161.1@KRBTEST.COM (short username kudu)
I20260810 03:26:03.492261 32114 webserver.cc:460] Webserver started at http://127.27.161.1:37071/ using document root <none> and password file <none>
I20260810 03:26:03.492993 32114 fs_manager.cc:359] Metadata directory not provided
I20260810 03:26:03.493129 32114 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:26:03.493462 32114 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:26:03.496635 32114 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786332336319544-28292-0/minicluster-data/ts-0/data/instance:
uuid: "f984d0e5e969450285e6823bceba3366"
format_stamp: "Formatted at 2026-08-10 03:26:03 on dist-test-slave-vpvm"
server_key: "a859727e0c78fe0a9979a5536d074f95"
server_key_iv: "b2f933fa9823e76849d8913798a20dc1"
server_key_version: "encryptionkey@0"
I20260810 03:26:03.497481 32114 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786332336319544-28292-0/minicluster-data/ts-0/wal/instance:
uuid: "f984d0e5e969450285e6823bceba3366"
format_stamp: "Formatted at 2026-08-10 03:26:03 on dist-test-slave-vpvm"
server_key: "a859727e0c78fe0a9979a5536d074f95"
server_key_iv: "b2f933fa9823e76849d8913798a20dc1"
server_key_version: "encryptionkey@0"
I20260810 03:26:03.503392 32114 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.002s
I20260810 03:26:03.507333 32137 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:26:03.508966 32114 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 03:26:03.509176 32114 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786332336319544-28292-0/minicluster-data/ts-0/data,/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786332336319544-28292-0/minicluster-data/ts-0/wal
uuid: "f984d0e5e969450285e6823bceba3366"
format_stamp: "Formatted at 2026-08-10 03:26:03 on dist-test-slave-vpvm"
server_key: "a859727e0c78fe0a9979a5536d074f95"
server_key_iv: "b2f933fa9823e76849d8913798a20dc1"
server_key_version: "encryptionkey@0"
I20260810 03:26:03.509411 32114 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786332336319544-28292-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786332336319544-28292-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786332336319544-28292-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:26:03.534063 32114 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:26:03.539628 32114 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:26:03.539952 32114 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:26:03.541081 32114 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:26:03.542666 32114 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:26:03.542752 32114 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:26:03.542891 32114 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:26:03.542945 32114 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:26:03.576846 32114 rpc_server.cc:307] RPC server started. Bound to: 127.27.161.1:36863
I20260810 03:26:03.576920 32321 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.161.1:36863 every 8 connection(s)
I20260810 03:26:03.578248 32114 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786332336319544-28292-0/minicluster-data/ts-0/data/info.pb
I20260810 03:26:03.580135 28292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iBH5/build/asan/bin/kudu as pid 32114
I20260810 03:26:03.580340 28292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786332336319544-28292-0/minicluster-data/ts-0/wal/instance
I20260810 03:26:03.581167 28292 external_mini_cluster.cc:1428] Setting key 827358542652d420b3538f79472d65bf
Aug 10 03:26:03 dist-test-slave-vpvm krb5kdc[31947](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332363, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.161.1@KRBTEST.COM for kudu/127.27.161.62@KRBTEST.COM
I20260810 03:26:03.602519 32327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:03.581181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.161.1:51955 (local address 127.27.161.62:33831)
0810 03:26:03.581702 (+ 521us) server_negotiation.cc:206] Beginning negotiation
0810 03:26:03.581706 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:26:03.583142 (+ 1436us) server_negotiation.cc:407] Connection header received
0810 03:26:03.585196 (+ 2054us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:26:03.585202 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:26:03.585301 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:26:03.585427 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:26:03.587351 (+ 1924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:03.588428 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:03.590368 (+ 1940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:03.590928 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:03.591076 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:03.595535 (+ 4459us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:26:03.595542 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:26:03.595545 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:26:03.595586 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:26:03.598502 (+ 2916us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:26:03.599633 (+ 1131us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:26:03.599640 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:26:03.599643 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:26:03.599790 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:26:03.600306 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:26:03.600309 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:26:03.600312 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:26:03.600611 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:26:03.600735 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0810 03:26:03.601884 (+ 1149us) server_negotiation.cc:299] Negotiation successful
0810 03:26:03.602133 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":411,"thread_start_us":268,"threads_started":1}
I20260810 03:26:03.604025 32328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:03.581998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:33831 (local address 127.27.161.1:51955)
0810 03:26:03.582931 (+ 933us) negotiation.cc:107] Waiting for socket to connect
0810 03:26:03.582991 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0810 03:26:03.584877 (+ 1886us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:26:03.585756 (+ 879us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:26:03.585766 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:26:03.586402 (+ 636us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:26:03.587139 (+ 737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:26:03.587160 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:26:03.588629 (+ 1469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:26:03.588634 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:26:03.590202 (+ 1568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:26:03.590218 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:26:03.591168 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:26:03.591174 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:26:03.591446 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:03.592245 (+ 799us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:26:03.592291 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:26:03.595215 (+ 2924us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:26:03.598696 (+ 3481us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:26:03.598707 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:26:03.598727 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:26:03.599456 (+ 729us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:26:03.599910 (+ 454us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:26:03.599914 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:26:03.599917 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:26:03.600198 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:26:03.600770 (+ 572us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:26:03.600780 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:26:03.601342 (+ 562us) client_negotiation.cc:769] Sending connection context
0810 03:26:03.601690 (+ 348us) client_negotiation.cc:240] Negotiation successful
0810 03:26:03.602387 (+ 697us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":629,"thread_start_us":357,"threads_started":1}
WARNING: no policy specified for kudu/127.27.161.2@KRBTEST.COM; defaulting to no policy
I20260810 03:26:03.606739 32323 heartbeater.cc:347] Connected to a master server at 127.27.161.62:33831
I20260810 03:26:03.607143 32323 heartbeater.cc:464] Registering TS with master...
I20260810 03:26:03.608000 32323 heartbeater.cc:511] Master 127.27.161.62:33831 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.161.2@KRBTEST.COM" created.
I20260810 03:26:03.610663 32023 ts_manager.cc:194] Registered new tserver with Master: f984d0e5e969450285e6823bceba3366 (127.27.161.1:36863)
I20260810 03:26:03.612197 32023 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.161.1@KRBTEST.COM'} at 127.27.161.1:51955
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.161.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.2.keytab.
Entry for principal kudu/127.27.161.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.2.keytab.
I20260810 03:26:03.637418 28292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iBH5/build/asan/bin/kudu
/tmp/dist-test-task24iBH5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786332336319544-28292-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786332336319544-28292-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786332336319544-28292-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786332336319544-28292-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.161.2:0
--local_ip_for_outbound_sockets=127.27.161.2
--webserver_interface=127.27.161.2
--webserver_port=0
--tserver_master_addrs=127.27.161.62:33831
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.2.keytab
--principal=kudu/127.27.161.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.161.20:43587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786332336319544-28292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786332336319544-28292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786332336319544-28292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:26:03.997498 32332 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:26:03.997838 32332 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:26:03.998013 32332 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:26:04.005563 32332 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:26:04.005728 32332 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:26:04.005895 32332 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.161.2
I20260810 03:26:04.016326 32332 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.161.20:43587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786332336319544-28292-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786332336319544-28292-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.2.keytab
--principal=kudu/127.27.161.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.161.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786332336319544-28292-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.161.2
--webserver_port=0
--tserver_master_addrs=127.27.161.62:33831
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.161.2
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786332336319544-28292-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:26:04.017757 32332 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:26:04.019218 32332 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:26:04.033182 32338 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:26:04.033143 32342 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:26:04.033111 32339 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:26:04.034297 32332 server_base.cc:1034] running on GCE node
I20260810 03:26:04.034992 32332 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:26:04.036262 32332 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:26:04.037530 32332 hybrid_clock.cc:648] HybridClock initialized: now 1786332364037501 us; error 73 us; skew 500 ppm
I20260810 03:26:04.037905 32332 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:26:04 dist-test-slave-vpvm krb5kdc[31947](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332364, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.161.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:26:04.041285 32332 init.cc:377] Logged in from keytab as kudu/127.27.161.2@KRBTEST.COM (short username kudu)
I20260810 03:26:04.043051 32332 webserver.cc:460] Webserver started at http://127.27.161.2:33947/ using document root <none> and password file <none>
I20260810 03:26:04.043807 32332 fs_manager.cc:359] Metadata directory not provided
I20260810 03:26:04.043913 32332 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:26:04.044217 32332 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:26:04.047425 32332 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786332336319544-28292-0/minicluster-data/ts-1/data/instance:
uuid: "b559b0c988c14a07906b5ce26c98b51b"
format_stamp: "Formatted at 2026-08-10 03:26:04 on dist-test-slave-vpvm"
server_key: "e983a3ff69ef810f214772d8d24cc3a4"
server_key_iv: "d6a8947c62179ba8ae0bf8fa0e1699f4"
server_key_version: "encryptionkey@0"
I20260810 03:26:04.048324 32332 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786332336319544-28292-0/minicluster-data/ts-1/wal/instance:
uuid: "b559b0c988c14a07906b5ce26c98b51b"
format_stamp: "Formatted at 2026-08-10 03:26:04 on dist-test-slave-vpvm"
server_key: "e983a3ff69ef810f214772d8d24cc3a4"
server_key_iv: "d6a8947c62179ba8ae0bf8fa0e1699f4"
server_key_version: "encryptionkey@0"
I20260810 03:26:04.054311 32332 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260810 03:26:04.058429 32350 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:26:04.060045 32332 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 03:26:04.060231 32332 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786332336319544-28292-0/minicluster-data/ts-1/data,/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786332336319544-28292-0/minicluster-data/ts-1/wal
uuid: "b559b0c988c14a07906b5ce26c98b51b"
format_stamp: "Formatted at 2026-08-10 03:26:04 on dist-test-slave-vpvm"
server_key: "e983a3ff69ef810f214772d8d24cc3a4"
server_key_iv: "d6a8947c62179ba8ae0bf8fa0e1699f4"
server_key_version: "encryptionkey@0"
I20260810 03:26:04.060441 32332 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786332336319544-28292-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786332336319544-28292-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786332336319544-28292-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:26:04.081616 32332 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:26:04.086159 32332 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:26:04.086488 32332 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:26:04.087661 32332 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:26:04.089178 32332 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:26:04.089264 32332 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:26:04.089397 32332 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:26:04.089452 32332 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:26:04.121759 32332 rpc_server.cc:307] RPC server started. Bound to: 127.27.161.2:46403
I20260810 03:26:04.121811 32522 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.161.2:46403 every 8 connection(s)
I20260810 03:26:04.123461 32332 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786332336319544-28292-0/minicluster-data/ts-1/data/info.pb
I20260810 03:26:04.133456 28292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iBH5/build/asan/bin/kudu as pid 32332
I20260810 03:26:04.133675 28292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786332336319544-28292-0/minicluster-data/ts-1/wal/instance
I20260810 03:26:04.134306 28292 external_mini_cluster.cc:1428] Setting key c3a989d543c5ab250b6d58f2f866e98e
Aug 10 03:26:04 dist-test-slave-vpvm krb5kdc[31947](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332364, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.161.2@KRBTEST.COM for kudu/127.27.161.62@KRBTEST.COM
I20260810 03:26:04.146621 32526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:04.126397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.161.2:43317 (local address 127.27.161.62:33831)
0810 03:26:04.126842 (+ 445us) server_negotiation.cc:206] Beginning negotiation
0810 03:26:04.126847 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:26:04.127967 (+ 1120us) server_negotiation.cc:407] Connection header received
0810 03:26:04.129903 (+ 1936us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:26:04.129909 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:26:04.130034 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:26:04.130139 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:26:04.132028 (+ 1889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:04.132770 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:04.135274 (+ 2504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:04.135764 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:04.135900 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:04.140242 (+ 4342us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:26:04.140248 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:26:04.140252 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:26:04.140290 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:26:04.142929 (+ 2639us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:26:04.143852 (+ 923us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:26:04.143856 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:26:04.143858 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:26:04.143982 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:26:04.144504 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:26:04.144509 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:26:04.144511 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:26:04.144828 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:26:04.144965 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0810 03:26:04.145935 (+ 970us) server_negotiation.cc:299] Negotiation successful
0810 03:26:04.146227 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":333,"thread_start_us":222,"threads_started":1}
I20260810 03:26:04.147470 32528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:04.126999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:33831 (local address 127.27.161.2:43317)
0810 03:26:04.127787 (+ 788us) negotiation.cc:107] Waiting for socket to connect
0810 03:26:04.127838 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0810 03:26:04.129590 (+ 1752us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:26:04.130421 (+ 831us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:26:04.130430 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:26:04.131154 (+ 724us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:26:04.131810 (+ 656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:26:04.131832 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:26:04.132943 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:26:04.132947 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:26:04.135093 (+ 2146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:26:04.135108 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:26:04.136025 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:26:04.136034 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:26:04.136276 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:04.136956 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:26:04.136998 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:26:04.139924 (+ 2926us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:26:04.143092 (+ 3168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:26:04.143102 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:26:04.143121 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:26:04.143741 (+ 620us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:26:04.144088 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:26:04.144092 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:26:04.144095 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:26:04.144356 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:26:04.144964 (+ 608us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:26:04.144973 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:26:04.145450 (+ 477us) client_negotiation.cc:769] Sending connection context
0810 03:26:04.145773 (+ 323us) client_negotiation.cc:240] Negotiation successful
0810 03:26:04.146280 (+ 507us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":525,"thread_start_us":292,"threads_started":1}
I20260810 03:26:04.149008 32523 heartbeater.cc:347] Connected to a master server at 127.27.161.62:33831
I20260810 03:26:04.149354 32523 heartbeater.cc:464] Registering TS with master...
I20260810 03:26:04.150225 32523 heartbeater.cc:511] Master 127.27.161.62:33831 requested a full tablet report, sending...
I20260810 03:26:04.151885 32023 ts_manager.cc:194] Registered new tserver with Master: b559b0c988c14a07906b5ce26c98b51b (127.27.161.2:46403)
I20260810 03:26:04.152750 32023 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.161.2@KRBTEST.COM'} at 127.27.161.2:43317
WARNING: no policy specified for kudu/127.27.161.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.161.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.161.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.3.keytab.
Entry for principal kudu/127.27.161.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.3.keytab.
I20260810 03:26:04.189793 28292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iBH5/build/asan/bin/kudu
/tmp/dist-test-task24iBH5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786332336319544-28292-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786332336319544-28292-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786332336319544-28292-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786332336319544-28292-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.161.3:0
--local_ip_for_outbound_sockets=127.27.161.3
--webserver_interface=127.27.161.3
--webserver_port=0
--tserver_master_addrs=127.27.161.62:33831
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.3.keytab
--principal=kudu/127.27.161.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.161.20:43587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786332336319544-28292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786332336319544-28292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786332336319544-28292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:26:04.542276 32531 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:26:04.542604 32531 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:26:04.542755 32531 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:26:04.550277 32531 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:26:04.550422 32531 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:26:04.550594 32531 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.161.3
I20260810 03:26:04.560899 32531 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.161.20:43587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786332336319544-28292-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786332336319544-28292-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.3.keytab
--principal=kudu/127.27.161.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.161.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786332336319544-28292-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.161.3
--webserver_port=0
--tserver_master_addrs=127.27.161.62:33831
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.161.3
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786332336319544-28292-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:26:04.562353 32531 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:26:04.563723 32531 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:26:04.577630 32539 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:26:04.577642 32543 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:26:04.578653 32531 server_base.cc:1034] running on GCE node
W20260810 03:26:04.582139 32541 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:26:04.582852 32531 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:26:04.583871 32531 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:26:04.585073 32531 hybrid_clock.cc:648] HybridClock initialized: now 1786332364585031 us; error 35 us; skew 500 ppm
I20260810 03:26:04.585398 32531 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:26:04 dist-test-slave-vpvm krb5kdc[31947](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332364, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.161.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:26:04.588740 32531 init.cc:377] Logged in from keytab as kudu/127.27.161.3@KRBTEST.COM (short username kudu)
I20260810 03:26:04.590360 32531 webserver.cc:460] Webserver started at http://127.27.161.3:33425/ using document root <none> and password file <none>
I20260810 03:26:04.591086 32531 fs_manager.cc:359] Metadata directory not provided
I20260810 03:26:04.591209 32531 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:26:04.591526 32531 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:26:04.594652 32531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786332336319544-28292-0/minicluster-data/ts-2/data/instance:
uuid: "ef6057e0cb394b7f8bffc630db50a376"
format_stamp: "Formatted at 2026-08-10 03:26:04 on dist-test-slave-vpvm"
server_key: "7c40b857acb71ca41782e80bbbdbceb4"
server_key_iv: "d8c3cdfab352d970f0bdfb4f9e5418fb"
server_key_version: "encryptionkey@0"
I20260810 03:26:04.595723 32531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786332336319544-28292-0/minicluster-data/ts-2/wal/instance:
uuid: "ef6057e0cb394b7f8bffc630db50a376"
format_stamp: "Formatted at 2026-08-10 03:26:04 on dist-test-slave-vpvm"
server_key: "7c40b857acb71ca41782e80bbbdbceb4"
server_key_iv: "d8c3cdfab352d970f0bdfb4f9e5418fb"
server_key_version: "encryptionkey@0"
I20260810 03:26:04.602007 32531 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.005s
I20260810 03:26:04.606153 32558 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:26:04.607652 32531 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260810 03:26:04.607842 32531 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786332336319544-28292-0/minicluster-data/ts-2/data,/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786332336319544-28292-0/minicluster-data/ts-2/wal
uuid: "ef6057e0cb394b7f8bffc630db50a376"
format_stamp: "Formatted at 2026-08-10 03:26:04 on dist-test-slave-vpvm"
server_key: "7c40b857acb71ca41782e80bbbdbceb4"
server_key_iv: "d8c3cdfab352d970f0bdfb4f9e5418fb"
server_key_version: "encryptionkey@0"
I20260810 03:26:04.608075 32531 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786332336319544-28292-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786332336319544-28292-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786332336319544-28292-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:26:04.615772 32323 heartbeater.cc:503] Master 127.27.161.62:33831 was elected leader, sending a full tablet report...
I20260810 03:26:04.619664 32531 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:26:04.623821 32531 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:26:04.624104 32531 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:26:04.625092 32531 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:26:04.626624 32531 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:26:04.626706 32531 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:26:04.626832 32531 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:26:04.626886 32531 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:26:04.658391 32531 rpc_server.cc:307] RPC server started. Bound to: 127.27.161.3:40113
I20260810 03:26:04.658432 32730 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.161.3:40113 every 8 connection(s)
I20260810 03:26:04.660048 32531 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786332336319544-28292-0/minicluster-data/ts-2/data/info.pb
I20260810 03:26:04.666705 28292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iBH5/build/asan/bin/kudu as pid 32531
I20260810 03:26:04.666925 28292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786332336319544-28292-0/minicluster-data/ts-2/wal/instance
I20260810 03:26:04.667363 28292 external_mini_cluster.cc:1428] Setting key 566a927d869d368e3da8c22191f1e49e
Aug 10 03:26:04 dist-test-slave-vpvm krb5kdc[31947](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332364, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.161.3@KRBTEST.COM for kudu/127.27.161.62@KRBTEST.COM
I20260810 03:26:04.684020 32733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:04.663032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.161.3:56019 (local address 127.27.161.62:33831)
0810 03:26:04.663542 (+ 510us) server_negotiation.cc:206] Beginning negotiation
0810 03:26:04.663547 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:26:04.664489 (+ 942us) server_negotiation.cc:407] Connection header received
0810 03:26:04.666491 (+ 2002us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:26:04.666497 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:26:04.666585 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:26:04.666685 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:26:04.668606 (+ 1921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:04.669571 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:04.671410 (+ 1839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:04.671835 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:04.671948 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:04.675710 (+ 3762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:26:04.675716 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:26:04.675720 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:26:04.675759 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:26:04.679512 (+ 3753us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:26:04.681073 (+ 1561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:26:04.681081 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:26:04.681083 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:26:04.681297 (+ 214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:26:04.681897 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:26:04.681901 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:26:04.681904 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:26:04.682294 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:26:04.682460 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0810 03:26:04.683329 (+ 869us) server_negotiation.cc:299] Negotiation successful
0810 03:26:04.683626 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":405,"thread_start_us":258,"threads_started":1}
I20260810 03:26:04.685050 32735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:04.663548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:33831 (local address 127.27.161.3:56019)
0810 03:26:04.664315 (+ 767us) negotiation.cc:107] Waiting for socket to connect
0810 03:26:04.664359 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0810 03:26:04.666176 (+ 1817us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:26:04.666974 (+ 798us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:26:04.666984 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:26:04.667655 (+ 671us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:26:04.668419 (+ 764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:26:04.668442 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:26:04.669742 (+ 1300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:26:04.669747 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:26:04.671247 (+ 1500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:26:04.671263 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:26:04.672082 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:26:04.672091 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:26:04.672343 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:04.673079 (+ 736us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:26:04.673104 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:26:04.675448 (+ 2344us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:26:04.679811 (+ 4363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:26:04.679824 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:26:04.679849 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:26:04.680818 (+ 969us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:26:04.681454 (+ 636us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:26:04.681458 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:26:04.681460 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:26:04.681755 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:26:04.682455 (+ 700us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:26:04.682465 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:26:04.682880 (+ 415us) client_negotiation.cc:769] Sending connection context
0810 03:26:04.683157 (+ 277us) client_negotiation.cc:240] Negotiation successful
0810 03:26:04.683829 (+ 672us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":528,"thread_start_us":292,"threads_started":1}
I20260810 03:26:04.686674 32731 heartbeater.cc:347] Connected to a master server at 127.27.161.62:33831
I20260810 03:26:04.687017 32731 heartbeater.cc:464] Registering TS with master...
I20260810 03:26:04.687777 32731 heartbeater.cc:511] Master 127.27.161.62:33831 requested a full tablet report, sending...
I20260810 03:26:04.689433 32023 ts_manager.cc:194] Registered new tserver with Master: ef6057e0cb394b7f8bffc630db50a376 (127.27.161.3:40113)
I20260810 03:26:04.690420 32023 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.161.3@KRBTEST.COM'} at 127.27.161.3:56019
I20260810 03:26:04.696305 28292 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260810 03:26:04.739519 32733 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:26:04.736388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35044 (local address 127.27.161.62:33831)
0810 03:26:04.736588 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0810 03:26:04.736594 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:26:04.737212 (+ 618us) server_negotiation.cc:407] Connection header received
0810 03:26:04.737334 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:26:04.737340 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:26:04.737459 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:26:04.737596 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:26:04.739213 (+ 1617us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35044: BlockingRecv error: recv got EOF from 127.0.0.1:35044 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
I20260810 03:26:04.742667 28292 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iBH5/build/asan/bin/kudu pid 32114
W20260810 03:26:04.745908 28292 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task24iBH5/build/asan/bin/kudu pid 32114: Not authorized: Client connection negotiation failed: client connection to 127.27.161.1:36863: server requires authentication, but client does not have Kerberos credentials available
I20260810 03:26:04.746181 28292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iBH5/build/asan/bin/kudu with pid 32114
I20260810 03:26:04.770305 28292 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iBH5/build/asan/bin/kudu pid 32332
W20260810 03:26:04.773504 28292 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task24iBH5/build/asan/bin/kudu pid 32332: Not authorized: Client connection negotiation failed: client connection to 127.27.161.2:46403: server requires authentication, but client does not have Kerberos credentials available
I20260810 03:26:04.773686 28292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iBH5/build/asan/bin/kudu with pid 32332
W20260810 03:26:04.773777 32749 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:26:04.771513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42946 (local address 127.27.161.2:46403)
0810 03:26:04.772195 (+ 682us) server_negotiation.cc:206] Beginning negotiation
0810 03:26:04.772204 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 03:26:04.772237 (+ 33us) server_negotiation.cc:407] Connection header received
0810 03:26:04.772312 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:26:04.772319 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:26:04.772524 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:26:04.772660 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:26:04.773443 (+ 783us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42946: BlockingRecv error: recv got EOF from 127.0.0.1:42946 (error 108)
Metrics: {"server-negotiator.queue_time_us":497,"thread_start_us":332,"threads_started":1}
I20260810 03:26:04.797308 28292 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iBH5/build/asan/bin/kudu pid 32531
W20260810 03:26:04.800846 28292 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task24iBH5/build/asan/bin/kudu pid 32531: Not authorized: Client connection negotiation failed: client connection to 127.27.161.3:40113: server requires authentication, but client does not have Kerberos credentials available
I20260810 03:26:04.801005 28292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iBH5/build/asan/bin/kudu with pid 32531
I20260810 03:26:04.827420 28292 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iBH5/build/asan/bin/kudu pid 31961
I20260810 03:26:04.828806 32061 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:26:04.935067 28292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iBH5/build/asan/bin/kudu with pid 31961
2026-08-10T03:26:04Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (4667 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786332336319544-28292-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 03:26:05 dist-test-slave-vpvm krb5kdc[32756](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 03:26:05 dist-test-slave-vpvm krb5kdc[32756](info): set up 2 sockets
Aug 10 03:26:05 dist-test-slave-vpvm krb5kdc[32756](info): commencing operation
krb5kdc: starting...
W20260810 03:26:06.157400 28292 mini_kdc.cc:121] Time spent starting KDC: real 1.168s user 0.000s sys 0.031s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786332336319544-28292-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786332336319544-28292-0/krb5kdc/test-user.keytab.
Aug 10 03:26:06 dist-test-slave-vpvm krb5kdc[32756](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332366, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T03:26:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T03:26:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.161.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.161.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.161.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.62.keytab.
Entry for principal kudu/127.27.161.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.62.keytab.
I20260810 03:26:06.410519 28292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iBH5/build/asan/bin/kudu
/tmp/dist-test-task24iBH5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786332336319544-28292-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786332336319544-28292-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786332336319544-28292-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786332336319544-28292-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.161.62:41231
--webserver_interface=127.27.161.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.62.keytab
--principal=kudu/127.27.161.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.161.20:43925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.161.62:41231
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786332336319544-28292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786332336319544-28292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786332336319544-28292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:26:06.759786 306 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:26:06.760113 306 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:26:06.760217 306 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:26:06.767645 306 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:26:06.767766 306 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:26:06.767813 306 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:26:06.767877 306 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:26:06.767925 306 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:26:06.778083 306 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.161.20:43925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786332336319544-28292-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786332336319544-28292-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.161.62:41231
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.62.keytab
--principal=kudu/127.27.161.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.161.62:41231
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786332336319544-28292-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.161.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786332336319544-28292-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:26:06.779451 306 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:26:06.780788 306 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:26:06.793362 313 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:26:06.793296 314 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:26:06.793390 316 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:26:06.794360 306 server_base.cc:1034] running on GCE node
I20260810 03:26:06.795212 306 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:26:06.796708 306 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:26:06.797915 306 hybrid_clock.cc:648] HybridClock initialized: now 1786332366797852 us; error 58 us; skew 500 ppm
I20260810 03:26:06.798316 306 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:26:06 dist-test-slave-vpvm krb5kdc[32756](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332366, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.161.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:26:06.801353 306 init.cc:377] Logged in from keytab as kudu/127.27.161.62@KRBTEST.COM (short username kudu)
I20260810 03:26:06.802927 306 webserver.cc:460] Webserver started at http://127.27.161.62:43443/ using document root <none> and password file <none>
I20260810 03:26:06.803656 306 fs_manager.cc:359] Metadata directory not provided
I20260810 03:26:06.803779 306 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:26:06.804090 306 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:26:06.807247 306 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786332336319544-28292-0/minicluster-data/master-0/data/instance:
uuid: "5154ed82c85b485ba46e18dd1692f7d8"
format_stamp: "Formatted at 2026-08-10 03:26:06 on dist-test-slave-vpvm"
server_key: "0cf26d481f791f26f44452f1f6ed4314"
server_key_iv: "f68d3701b5823b22199caf130fe1ef27"
server_key_version: "encryptionkey@0"
I20260810 03:26:06.808080 306 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786332336319544-28292-0/minicluster-data/master-0/wal/instance:
uuid: "5154ed82c85b485ba46e18dd1692f7d8"
format_stamp: "Formatted at 2026-08-10 03:26:06 on dist-test-slave-vpvm"
server_key: "0cf26d481f791f26f44452f1f6ed4314"
server_key_iv: "f68d3701b5823b22199caf130fe1ef27"
server_key_version: "encryptionkey@0"
I20260810 03:26:06.813598 306 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260810 03:26:06.817634 328 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:26:06.819114 306 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 03:26:06.819305 306 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786332336319544-28292-0/minicluster-data/master-0/data,/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786332336319544-28292-0/minicluster-data/master-0/wal
uuid: "5154ed82c85b485ba46e18dd1692f7d8"
format_stamp: "Formatted at 2026-08-10 03:26:06 on dist-test-slave-vpvm"
server_key: "0cf26d481f791f26f44452f1f6ed4314"
server_key_iv: "f68d3701b5823b22199caf130fe1ef27"
server_key_version: "encryptionkey@0"
I20260810 03:26:06.819516 306 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786332336319544-28292-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786332336319544-28292-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786332336319544-28292-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:26:06.836151 306 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:26:06.841301 306 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:26:06.841611 306 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:26:06.860795 306 rpc_server.cc:307] RPC server started. Bound to: 127.27.161.62:41231
I20260810 03:26:06.860821 401 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.161.62:41231 every 8 connection(s)
I20260810 03:26:06.862278 306 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786332336319544-28292-0/minicluster-data/master-0/data/info.pb
I20260810 03:26:06.866225 402 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:26:06.866532 28292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iBH5/build/asan/bin/kudu as pid 306
I20260810 03:26:06.866721 28292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786332336319544-28292-0/minicluster-data/master-0/wal/instance
I20260810 03:26:06.867367 28292 external_mini_cluster.cc:1428] Setting key 26d847623553350cde6e78dbdcc7693e
I20260810 03:26:06.880558 402 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5154ed82c85b485ba46e18dd1692f7d8: Bootstrap starting.
Aug 10 03:26:06 dist-test-slave-vpvm krb5kdc[32756](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332366, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.161.62@KRBTEST.COM
I20260810 03:26:06.884918 402 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5154ed82c85b485ba46e18dd1692f7d8: Neither blocks nor log segments found. Creating new log.
I20260810 03:26:06.886238 402 log.cc:824] T 00000000000000000000000000000000 P 5154ed82c85b485ba46e18dd1692f7d8: Log is configured to *not* fsync() on all Append() calls
I20260810 03:26:06.889555 402 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5154ed82c85b485ba46e18dd1692f7d8: No bootstrap required, opened a new log
I20260810 03:26:06.893724 407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:06.869143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33044 (local address 127.27.161.62:41231)
0810 03:26:06.869973 (+ 830us) server_negotiation.cc:206] Beginning negotiation
0810 03:26:06.869989 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 03:26:06.870042 (+ 53us) server_negotiation.cc:407] Connection header received
0810 03:26:06.872357 (+ 2315us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:26:06.872385 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:26:06.872867 (+ 482us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:26:06.873238 (+ 371us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:26:06.875245 (+ 2007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:06.877180 (+ 1935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:06.879180 (+ 2000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:06.880153 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:06.880314 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:06.885525 (+ 5211us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:26:06.885537 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:26:06.885555 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:26:06.885615 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:26:06.889246 (+ 3631us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:26:06.890204 (+ 958us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:26:06.890212 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:26:06.890219 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:26:06.890411 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:26:06.890813 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:26:06.890816 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:26:06.890818 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:26:06.891432 (+ 614us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:26:06.891691 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0810 03:26:06.892167 (+ 476us) server_negotiation.cc:299] Negotiation successful
0810 03:26:06.892595 (+ 428us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":586,"thread_start_us":366,"threads_started":1}
I20260810 03:26:06.897888 28292 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 03:26:06.898420 402 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5154ed82c85b485ba46e18dd1692f7d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5154ed82c85b485ba46e18dd1692f7d8" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 41231 } }
I20260810 03:26:06.898900 402 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5154ed82c85b485ba46e18dd1692f7d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:26:06.898998 402 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5154ed82c85b485ba46e18dd1692f7d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5154ed82c85b485ba46e18dd1692f7d8, State: Initialized, Role: FOLLOWER
I20260810 03:26:06.899643 402 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5154ed82c85b485ba46e18dd1692f7d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5154ed82c85b485ba46e18dd1692f7d8" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 41231 } }
I20260810 03:26:06.899905 402 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5154ed82c85b485ba46e18dd1692f7d8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 03:26:06.900036 402 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5154ed82c85b485ba46e18dd1692f7d8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 03:26:06.900208 402 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5154ed82c85b485ba46e18dd1692f7d8 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:26:06.903362 402 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5154ed82c85b485ba46e18dd1692f7d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5154ed82c85b485ba46e18dd1692f7d8" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 41231 } }
I20260810 03:26:06.904021 402 leader_election.cc:302] T 00000000000000000000000000000000 P 5154ed82c85b485ba46e18dd1692f7d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5154ed82c85b485ba46e18dd1692f7d8; no voters:
I20260810 03:26:06.904716 402 leader_election.cc:288] T 00000000000000000000000000000000 P 5154ed82c85b485ba46e18dd1692f7d8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 03:26:06.904812 409 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5154ed82c85b485ba46e18dd1692f7d8 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:26:06.905918 409 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5154ed82c85b485ba46e18dd1692f7d8 [term 1 LEADER]: Becoming Leader. State: Replica: 5154ed82c85b485ba46e18dd1692f7d8, State: Running, Role: LEADER
I20260810 03:26:06.906499 409 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5154ed82c85b485ba46e18dd1692f7d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5154ed82c85b485ba46e18dd1692f7d8" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 41231 } }
I20260810 03:26:06.907030 402 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5154ed82c85b485ba46e18dd1692f7d8 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:26:06.909274 411 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5154ed82c85b485ba46e18dd1692f7d8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5154ed82c85b485ba46e18dd1692f7d8. Latest consensus state: current_term: 1 leader_uuid: "5154ed82c85b485ba46e18dd1692f7d8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5154ed82c85b485ba46e18dd1692f7d8" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 41231 } } }
I20260810 03:26:06.909219 410 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5154ed82c85b485ba46e18dd1692f7d8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5154ed82c85b485ba46e18dd1692f7d8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5154ed82c85b485ba46e18dd1692f7d8" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 41231 } } }
I20260810 03:26:06.909693 410 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5154ed82c85b485ba46e18dd1692f7d8 [sys.catalog]: This master's current role is: LEADER
I20260810 03:26:06.909693 411 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5154ed82c85b485ba46e18dd1692f7d8 [sys.catalog]: This master's current role is: LEADER
I20260810 03:26:06.911190 419 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 03:26:06.930035 419 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 03:26:06.947410 419 catalog_manager.cc:1347] Generated new cluster ID: 0d0a325bef05463d85be188cf4785bf7
I20260810 03:26:06.947543 419 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 03:26:06.960817 419 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 03:26:06.962908 419 catalog_manager.cc:1504] Loading token signing keys...
I20260810 03:26:06.963305 437 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260810 03:26:06.972322 419 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5154ed82c85b485ba46e18dd1692f7d8: Generated new TSK 0
I20260810 03:26:06.973448 419 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 03:26:06.975660 407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:06.966458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33050 (local address 127.27.161.62:41231)
0810 03:26:06.966638 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0810 03:26:06.966645 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:26:06.967039 (+ 394us) server_negotiation.cc:407] Connection header received
0810 03:26:06.967238 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:26:06.967245 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:26:06.967336 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:26:06.967455 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:26:06.968620 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:06.969741 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:06.971250 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:06.971966 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:06.972117 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:06.973556 (+ 1439us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:26:06.973564 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:26:06.973569 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:26:06.974754 (+ 1185us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:26:06.974843 (+ 89us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:26:06.974886 (+ 43us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:26:06.975015 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0810 03:26:06.975133 (+ 118us) server_negotiation.cc:299] Negotiation successful
0810 03:26:06.975256 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260810 03:26:06.985292 28292 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iBH5/build/asan/bin/kudu pid 306
I20260810 03:26:06.986982 395 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:26:07.080587 28292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iBH5/build/asan/bin/kudu with pid 306
2026-08-10T03:26:07Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (2136 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786332336319544-28292-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 03:26:07 dist-test-slave-vpvm krb5kdc[447](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 03:26:07 dist-test-slave-vpvm krb5kdc[447](info): set up 2 sockets
Aug 10 03:26:07 dist-test-slave-vpvm krb5kdc[447](info): commencing operation
krb5kdc: starting...
W20260810 03:26:08.292692 28292 mini_kdc.cc:121] Time spent starting KDC: real 1.167s user 0.000s sys 0.032s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786332336319544-28292-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786332336319544-28292-0/krb5kdc/test-user.keytab.
Aug 10 03:26:08 dist-test-slave-vpvm krb5kdc[447](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332368, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T03:26:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T03:26:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.161.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.161.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.161.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.62.keytab.
Entry for principal kudu/127.27.161.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.62.keytab.
I20260810 03:26:08.541196 28292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iBH5/build/asan/bin/kudu
/tmp/dist-test-task24iBH5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786332336319544-28292-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786332336319544-28292-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786332336319544-28292-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786332336319544-28292-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.161.62:36693
--webserver_interface=127.27.161.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.62.keytab
--principal=kudu/127.27.161.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.161.20:42417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.161.62:36693
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786332336319544-28292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786332336319544-28292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786332336319544-28292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:26:08.890729 465 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:26:08.891067 465 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:26:08.891188 465 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:26:08.898612 465 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:26:08.898734 465 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:26:08.898808 465 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:26:08.898887 465 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:26:08.898947 465 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:26:08.909089 465 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.161.20:42417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786332336319544-28292-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786332336319544-28292-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.161.62:36693
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.62.keytab
--principal=kudu/127.27.161.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.161.62:36693
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786332336319544-28292-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.161.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786332336319544-28292-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:26:08.910532 465 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:26:08.911885 465 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:26:08.923614 474 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:26:08.923746 470 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:26:08.923687 476 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:26:08.924635 465 server_base.cc:1034] running on GCE node
I20260810 03:26:08.925487 465 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:26:08.927068 465 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:26:08.928303 465 hybrid_clock.cc:648] HybridClock initialized: now 1786332368928243 us; error 49 us; skew 500 ppm
I20260810 03:26:08.928714 465 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:26:08 dist-test-slave-vpvm krb5kdc[447](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332368, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.161.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:26:08.932044 465 init.cc:377] Logged in from keytab as kudu/127.27.161.62@KRBTEST.COM (short username kudu)
I20260810 03:26:08.933805 465 webserver.cc:460] Webserver started at http://127.27.161.62:33577/ using document root <none> and password file <none>
I20260810 03:26:08.934587 465 fs_manager.cc:359] Metadata directory not provided
I20260810 03:26:08.934718 465 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:26:08.935066 465 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:26:08.938277 465 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786332336319544-28292-0/minicluster-data/master-0/data/instance:
uuid: "2da0f84b7974452abc71cf5200c77d2e"
format_stamp: "Formatted at 2026-08-10 03:26:08 on dist-test-slave-vpvm"
server_key: "d318f6aa3233f9f4e11787ce9d81260b"
server_key_iv: "d70eca08c7bf8e7a346e7d63aff04395"
server_key_version: "encryptionkey@0"
I20260810 03:26:08.939152 465 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786332336319544-28292-0/minicluster-data/master-0/wal/instance:
uuid: "2da0f84b7974452abc71cf5200c77d2e"
format_stamp: "Formatted at 2026-08-10 03:26:08 on dist-test-slave-vpvm"
server_key: "d318f6aa3233f9f4e11787ce9d81260b"
server_key_iv: "d70eca08c7bf8e7a346e7d63aff04395"
server_key_version: "encryptionkey@0"
I20260810 03:26:08.944835 465 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260810 03:26:08.948717 487 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:26:08.950281 465 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 03:26:08.950484 465 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786332336319544-28292-0/minicluster-data/master-0/data,/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786332336319544-28292-0/minicluster-data/master-0/wal
uuid: "2da0f84b7974452abc71cf5200c77d2e"
format_stamp: "Formatted at 2026-08-10 03:26:08 on dist-test-slave-vpvm"
server_key: "d318f6aa3233f9f4e11787ce9d81260b"
server_key_iv: "d70eca08c7bf8e7a346e7d63aff04395"
server_key_version: "encryptionkey@0"
I20260810 03:26:08.950703 465 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786332336319544-28292-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786332336319544-28292-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786332336319544-28292-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:26:08.973881 465 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:26:08.978505 465 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:26:08.978814 465 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:26:08.998608 465 rpc_server.cc:307] RPC server started. Bound to: 127.27.161.62:36693
I20260810 03:26:08.998651 559 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.161.62:36693 every 8 connection(s)
I20260810 03:26:08.999969 465 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786332336319544-28292-0/minicluster-data/master-0/data/info.pb
I20260810 03:26:09.003823 561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:26:09.006738 28292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iBH5/build/asan/bin/kudu as pid 465
I20260810 03:26:09.006930 28292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786332336319544-28292-0/minicluster-data/master-0/wal/instance
I20260810 03:26:09.007556 28292 external_mini_cluster.cc:1428] Setting key f932dc801819d3decb3dade4b7ab0c21
I20260810 03:26:09.016320 561 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2da0f84b7974452abc71cf5200c77d2e: Bootstrap starting.
I20260810 03:26:09.020372 561 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2da0f84b7974452abc71cf5200c77d2e: Neither blocks nor log segments found. Creating new log.
I20260810 03:26:09.021829 561 log.cc:824] T 00000000000000000000000000000000 P 2da0f84b7974452abc71cf5200c77d2e: Log is configured to *not* fsync() on all Append() calls
Aug 10 03:26:09 dist-test-slave-vpvm krb5kdc[447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332368, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.161.62@KRBTEST.COM
I20260810 03:26:09.025285 561 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2da0f84b7974452abc71cf5200c77d2e: No bootstrap required, opened a new log
I20260810 03:26:09.031464 564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:09.009606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39352 (local address 127.27.161.62:36693)
0810 03:26:09.010371 (+ 765us) server_negotiation.cc:206] Beginning negotiation
0810 03:26:09.010383 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 03:26:09.010424 (+ 41us) server_negotiation.cc:407] Connection header received
0810 03:26:09.012108 (+ 1684us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:26:09.012132 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:26:09.012549 (+ 417us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:26:09.012884 (+ 335us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:26:09.014823 (+ 1939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:09.016170 (+ 1347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:09.017900 (+ 1730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:09.018763 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:09.018921 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:09.023867 (+ 4946us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:26:09.023875 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:26:09.023893 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:26:09.023937 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:26:09.027091 (+ 3154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:26:09.027951 (+ 860us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:26:09.027958 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:26:09.027967 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:26:09.028154 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:26:09.028577 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:26:09.028581 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:26:09.028582 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:26:09.029154 (+ 572us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:26:09.029396 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0810 03:26:09.029872 (+ 476us) server_negotiation.cc:299] Negotiation successful
0810 03:26:09.030354 (+ 482us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":553,"thread_start_us":339,"threads_started":1}
I20260810 03:26:09.034695 561 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2da0f84b7974452abc71cf5200c77d2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2da0f84b7974452abc71cf5200c77d2e" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 36693 } }
I20260810 03:26:09.035095 561 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2da0f84b7974452abc71cf5200c77d2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:26:09.035200 561 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2da0f84b7974452abc71cf5200c77d2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2da0f84b7974452abc71cf5200c77d2e, State: Initialized, Role: FOLLOWER
I20260810 03:26:09.035744 561 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2da0f84b7974452abc71cf5200c77d2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2da0f84b7974452abc71cf5200c77d2e" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 36693 } }
I20260810 03:26:09.035975 561 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2da0f84b7974452abc71cf5200c77d2e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 03:26:09.036109 561 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2da0f84b7974452abc71cf5200c77d2e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 03:26:09.036269 561 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2da0f84b7974452abc71cf5200c77d2e [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:26:09.038718 561 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2da0f84b7974452abc71cf5200c77d2e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2da0f84b7974452abc71cf5200c77d2e" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 36693 } }
I20260810 03:26:09.039250 561 leader_election.cc:302] T 00000000000000000000000000000000 P 2da0f84b7974452abc71cf5200c77d2e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2da0f84b7974452abc71cf5200c77d2e; no voters:
I20260810 03:26:09.039829 561 leader_election.cc:288] T 00000000000000000000000000000000 P 2da0f84b7974452abc71cf5200c77d2e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 03:26:09.039963 569 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2da0f84b7974452abc71cf5200c77d2e [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:26:09.040943 569 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2da0f84b7974452abc71cf5200c77d2e [term 1 LEADER]: Becoming Leader. State: Replica: 2da0f84b7974452abc71cf5200c77d2e, State: Running, Role: LEADER
I20260810 03:26:09.041617 569 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2da0f84b7974452abc71cf5200c77d2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2da0f84b7974452abc71cf5200c77d2e" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 36693 } }
I20260810 03:26:09.042140 561 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2da0f84b7974452abc71cf5200c77d2e [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:26:09.044551 570 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2da0f84b7974452abc71cf5200c77d2e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2da0f84b7974452abc71cf5200c77d2e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2da0f84b7974452abc71cf5200c77d2e" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 36693 } } }
I20260810 03:26:09.044612 571 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2da0f84b7974452abc71cf5200c77d2e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2da0f84b7974452abc71cf5200c77d2e. Latest consensus state: current_term: 1 leader_uuid: "2da0f84b7974452abc71cf5200c77d2e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2da0f84b7974452abc71cf5200c77d2e" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 36693 } } }
I20260810 03:26:09.044986 570 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2da0f84b7974452abc71cf5200c77d2e [sys.catalog]: This master's current role is: LEADER
I20260810 03:26:09.045080 571 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2da0f84b7974452abc71cf5200c77d2e [sys.catalog]: This master's current role is: LEADER
I20260810 03:26:09.047904 578 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 03:26:09.053028 578 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 03:26:09.063280 578 catalog_manager.cc:1347] Generated new cluster ID: 7369f44c4afa4b30bc615f1e5e24288a
I20260810 03:26:09.063412 578 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 03:26:09.078223 578 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 03:26:09.080209 578 catalog_manager.cc:1504] Loading token signing keys...
I20260810 03:26:09.092317 578 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2da0f84b7974452abc71cf5200c77d2e: Generated new TSK 0
I20260810 03:26:09.093534 578 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.161.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.161.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.161.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.1.keytab.
Entry for principal kudu/127.27.161.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.1.keytab.
I20260810 03:26:09.157238 28292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iBH5/build/asan/bin/kudu
/tmp/dist-test-task24iBH5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786332336319544-28292-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786332336319544-28292-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786332336319544-28292-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786332336319544-28292-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.161.1:0
--local_ip_for_outbound_sockets=127.27.161.1
--webserver_interface=127.27.161.1
--webserver_port=0
--tserver_master_addrs=127.27.161.62:36693
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.1.keytab
--principal=kudu/127.27.161.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.161.20:42417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786332336319544-28292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786332336319544-28292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786332336319544-28292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:26:09.524987 597 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:26:09.525333 597 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:26:09.525470 597 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:26:09.533170 597 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:26:09.533330 597 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:26:09.533492 597 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.161.1
I20260810 03:26:09.544068 597 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.161.20:42417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786332336319544-28292-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786332336319544-28292-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.1.keytab
--principal=kudu/127.27.161.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.161.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786332336319544-28292-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.161.1
--webserver_port=0
--tserver_master_addrs=127.27.161.62:36693
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.161.1
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786332336319544-28292-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:26:09.545472 597 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:26:09.547122 597 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:26:09.561894 604 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:26:09.561862 607 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:26:09.561987 603 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:26:09.562997 597 server_base.cc:1034] running on GCE node
I20260810 03:26:09.563717 597 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:26:09.564788 597 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:26:09.566054 597 hybrid_clock.cc:648] HybridClock initialized: now 1786332369566002 us; error 45 us; skew 500 ppm
I20260810 03:26:09.566397 597 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:26:09 dist-test-slave-vpvm krb5kdc[447](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332369, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.161.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:26:09.569726 597 init.cc:377] Logged in from keytab as kudu/127.27.161.1@KRBTEST.COM (short username kudu)
I20260810 03:26:09.571374 597 webserver.cc:460] Webserver started at http://127.27.161.1:42819/ using document root <none> and password file <none>
I20260810 03:26:09.572129 597 fs_manager.cc:359] Metadata directory not provided
I20260810 03:26:09.572259 597 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:26:09.572590 597 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:26:09.576403 597 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786332336319544-28292-0/minicluster-data/ts-0/data/instance:
uuid: "db9565b985364f68aa2ef4fd044dac07"
format_stamp: "Formatted at 2026-08-10 03:26:09 on dist-test-slave-vpvm"
server_key: "0d02801e7d6663a2aeb697279429db88"
server_key_iv: "76b5747394a43aa6396a01a132046b48"
server_key_version: "encryptionkey@0"
I20260810 03:26:09.577766 597 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786332336319544-28292-0/minicluster-data/ts-0/wal/instance:
uuid: "db9565b985364f68aa2ef4fd044dac07"
format_stamp: "Formatted at 2026-08-10 03:26:09 on dist-test-slave-vpvm"
server_key: "0d02801e7d6663a2aeb697279429db88"
server_key_iv: "76b5747394a43aa6396a01a132046b48"
server_key_version: "encryptionkey@0"
I20260810 03:26:09.584545 597 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260810 03:26:09.588850 618 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:26:09.590564 597 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260810 03:26:09.590767 597 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786332336319544-28292-0/minicluster-data/ts-0/data,/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786332336319544-28292-0/minicluster-data/ts-0/wal
uuid: "db9565b985364f68aa2ef4fd044dac07"
format_stamp: "Formatted at 2026-08-10 03:26:09 on dist-test-slave-vpvm"
server_key: "0d02801e7d6663a2aeb697279429db88"
server_key_iv: "76b5747394a43aa6396a01a132046b48"
server_key_version: "encryptionkey@0"
I20260810 03:26:09.590991 597 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786332336319544-28292-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786332336319544-28292-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786332336319544-28292-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:26:09.609840 597 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:26:09.614197 597 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:26:09.614511 597 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:26:09.615507 597 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:26:09.616930 597 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:26:09.617010 597 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:26:09.617143 597 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:26:09.617199 597 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:26:09.648957 597 rpc_server.cc:307] RPC server started. Bound to: 127.27.161.1:37295
I20260810 03:26:09.649012 776 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.161.1:37295 every 8 connection(s)
I20260810 03:26:09.650303 597 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786332336319544-28292-0/minicluster-data/ts-0/data/info.pb
I20260810 03:26:09.653642 28292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iBH5/build/asan/bin/kudu as pid 597
I20260810 03:26:09.653846 28292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786332336319544-28292-0/minicluster-data/ts-0/wal/instance
I20260810 03:26:09.654686 28292 external_mini_cluster.cc:1428] Setting key 2728aa34574c4988849cbd0dbe03f1a2
Aug 10 03:26:09 dist-test-slave-vpvm krb5kdc[447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332369, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.161.1@KRBTEST.COM for kudu/127.27.161.62@KRBTEST.COM
I20260810 03:26:09.672693 779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:09.653030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.161.1:45023 (local address 127.27.161.62:36693)
0810 03:26:09.653564 (+ 534us) server_negotiation.cc:206] Beginning negotiation
0810 03:26:09.653570 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:26:09.654710 (+ 1140us) server_negotiation.cc:407] Connection header received
0810 03:26:09.657035 (+ 2325us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:26:09.657039 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:26:09.657104 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:26:09.657193 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:26:09.659217 (+ 2024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:09.659936 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:09.662009 (+ 2073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:09.662424 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:09.662524 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:09.666490 (+ 3966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:26:09.666496 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:26:09.666501 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:26:09.666543 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:26:09.669446 (+ 2903us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:26:09.670342 (+ 896us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:26:09.670348 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:26:09.670351 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:26:09.670496 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:26:09.670949 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:26:09.670953 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:26:09.670956 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:26:09.671300 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:26:09.671441 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0810 03:26:09.672141 (+ 700us) server_negotiation.cc:299] Negotiation successful
0810 03:26:09.672355 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":394,"thread_start_us":253,"threads_started":1}
I20260810 03:26:09.673519 782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:09.653588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:36693 (local address 127.27.161.1:45023)
0810 03:26:09.654522 (+ 934us) negotiation.cc:107] Waiting for socket to connect
0810 03:26:09.654587 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0810 03:26:09.656739 (+ 2152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:26:09.657473 (+ 734us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:26:09.657484 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:26:09.658212 (+ 728us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:26:09.659000 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:26:09.659027 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:26:09.660108 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:26:09.660113 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:26:09.661841 (+ 1728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:26:09.661857 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:26:09.662654 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:26:09.662661 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:26:09.662933 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:09.663640 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:26:09.663668 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:26:09.666257 (+ 2589us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:26:09.669621 (+ 3364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:26:09.669630 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:26:09.669645 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:26:09.670200 (+ 555us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:26:09.670628 (+ 428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:26:09.670632 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:26:09.670635 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:26:09.670833 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:26:09.671408 (+ 575us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:26:09.671413 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:26:09.671746 (+ 333us) client_negotiation.cc:769] Sending connection context
0810 03:26:09.671974 (+ 228us) client_negotiation.cc:240] Negotiation successful
0810 03:26:09.672386 (+ 412us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":601,"thread_start_us":363,"threads_started":1}
I20260810 03:26:09.675125 777 heartbeater.cc:347] Connected to a master server at 127.27.161.62:36693
I20260810 03:26:09.675431 777 heartbeater.cc:464] Registering TS with master...
I20260810 03:26:09.676285 777 heartbeater.cc:511] Master 127.27.161.62:36693 requested a full tablet report, sending...
I20260810 03:26:09.678994 510 ts_manager.cc:194] Registered new tserver with Master: db9565b985364f68aa2ef4fd044dac07 (127.27.161.1:37295)
WARNING: no policy specified for kudu/127.27.161.2@KRBTEST.COM; defaulting to no policy
I20260810 03:26:09.680860 510 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.161.1@KRBTEST.COM'} at 127.27.161.1:45023
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.161.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.161.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.2.keytab.
Entry for principal kudu/127.27.161.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.2.keytab.
I20260810 03:26:09.711109 28292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iBH5/build/asan/bin/kudu
/tmp/dist-test-task24iBH5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786332336319544-28292-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786332336319544-28292-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786332336319544-28292-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786332336319544-28292-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.161.2:0
--local_ip_for_outbound_sockets=127.27.161.2
--webserver_interface=127.27.161.2
--webserver_port=0
--tserver_master_addrs=127.27.161.62:36693
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.2.keytab
--principal=kudu/127.27.161.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.161.20:42417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786332336319544-28292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786332336319544-28292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786332336319544-28292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:26:10.067436 788 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:26:10.067764 788 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:26:10.067911 788 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:26:10.075584 788 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:26:10.075731 788 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:26:10.075901 788 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.161.2
I20260810 03:26:10.086089 788 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.161.20:42417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786332336319544-28292-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786332336319544-28292-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.2.keytab
--principal=kudu/127.27.161.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.161.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786332336319544-28292-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.161.2
--webserver_port=0
--tserver_master_addrs=127.27.161.62:36693
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.161.2
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786332336319544-28292-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:26:10.087487 788 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:26:10.088910 788 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:26:10.102787 797 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:26:10.102799 800 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:26:10.102708 794 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:26:10.104080 788 server_base.cc:1034] running on GCE node
I20260810 03:26:10.104780 788 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:26:10.105887 788 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:26:10.107151 788 hybrid_clock.cc:648] HybridClock initialized: now 1786332370107092 us; error 45 us; skew 500 ppm
I20260810 03:26:10.107514 788 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:26:10 dist-test-slave-vpvm krb5kdc[447](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332370, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.161.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:26:10.113915 788 init.cc:377] Logged in from keytab as kudu/127.27.161.2@KRBTEST.COM (short username kudu)
I20260810 03:26:10.115741 788 webserver.cc:460] Webserver started at http://127.27.161.2:45483/ using document root <none> and password file <none>
I20260810 03:26:10.116686 788 fs_manager.cc:359] Metadata directory not provided
I20260810 03:26:10.116838 788 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:26:10.117187 788 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:26:10.120478 788 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786332336319544-28292-0/minicluster-data/ts-1/data/instance:
uuid: "0165a059953946f6aabc53cb2aa727e2"
format_stamp: "Formatted at 2026-08-10 03:26:10 on dist-test-slave-vpvm"
server_key: "761a8b6d820eaccbc04126dd8264d4ef"
server_key_iv: "9ab6541def10447847d944dddddc8a82"
server_key_version: "encryptionkey@0"
I20260810 03:26:10.121457 788 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786332336319544-28292-0/minicluster-data/ts-1/wal/instance:
uuid: "0165a059953946f6aabc53cb2aa727e2"
format_stamp: "Formatted at 2026-08-10 03:26:10 on dist-test-slave-vpvm"
server_key: "761a8b6d820eaccbc04126dd8264d4ef"
server_key_iv: "9ab6541def10447847d944dddddc8a82"
server_key_version: "encryptionkey@0"
I20260810 03:26:10.127185 788 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260810 03:26:10.130964 811 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:26:10.132395 788 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 03:26:10.132580 788 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786332336319544-28292-0/minicluster-data/ts-1/data,/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786332336319544-28292-0/minicluster-data/ts-1/wal
uuid: "0165a059953946f6aabc53cb2aa727e2"
format_stamp: "Formatted at 2026-08-10 03:26:10 on dist-test-slave-vpvm"
server_key: "761a8b6d820eaccbc04126dd8264d4ef"
server_key_iv: "9ab6541def10447847d944dddddc8a82"
server_key_version: "encryptionkey@0"
I20260810 03:26:10.132794 788 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786332336319544-28292-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786332336319544-28292-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786332336319544-28292-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:26:10.149937 788 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:26:10.154451 788 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:26:10.154753 788 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:26:10.155854 788 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:26:10.157589 788 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:26:10.157675 788 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:26:10.157805 788 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:26:10.157860 788 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:26:10.190182 788 rpc_server.cc:307] RPC server started. Bound to: 127.27.161.2:37641
I20260810 03:26:10.190251 981 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.161.2:37641 every 8 connection(s)
I20260810 03:26:10.191496 788 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786332336319544-28292-0/minicluster-data/ts-1/data/info.pb
I20260810 03:26:10.197422 28292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iBH5/build/asan/bin/kudu as pid 788
I20260810 03:26:10.197593 28292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786332336319544-28292-0/minicluster-data/ts-1/wal/instance
I20260810 03:26:10.198197 28292 external_mini_cluster.cc:1428] Setting key 5c30a147a82486e1ea6b0cf7a84efec5
Aug 10 03:26:10 dist-test-slave-vpvm krb5kdc[447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332370, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.161.2@KRBTEST.COM for kudu/127.27.161.62@KRBTEST.COM
I20260810 03:26:10.215106 986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:10.194128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.161.2:35883 (local address 127.27.161.62:36693)
0810 03:26:10.194597 (+ 469us) server_negotiation.cc:206] Beginning negotiation
0810 03:26:10.194602 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:26:10.195576 (+ 974us) server_negotiation.cc:407] Connection header received
0810 03:26:10.197501 (+ 1925us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:26:10.197507 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:26:10.197611 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:26:10.197717 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:26:10.199763 (+ 2046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:10.200664 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:10.203049 (+ 2385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:10.203506 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:10.203624 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:10.208231 (+ 4607us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:26:10.208238 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:26:10.208241 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:26:10.208283 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:26:10.211564 (+ 3281us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:26:10.212453 (+ 889us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:26:10.212460 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:26:10.212464 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:26:10.212638 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:26:10.213133 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:26:10.213137 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:26:10.213140 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:26:10.213485 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:26:10.213649 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0810 03:26:10.214521 (+ 872us) server_negotiation.cc:299] Negotiation successful
0810 03:26:10.214752 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317,"thread_start_us":205,"threads_started":1}
I20260810 03:26:10.216040 987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:10.194644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:36693 (local address 127.27.161.2:35883)
0810 03:26:10.195398 (+ 754us) negotiation.cc:107] Waiting for socket to connect
0810 03:26:10.195445 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0810 03:26:10.197220 (+ 1775us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:26:10.198057 (+ 837us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:26:10.198070 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:26:10.198861 (+ 791us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:26:10.199573 (+ 712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:26:10.199594 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:26:10.200850 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:26:10.200855 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:26:10.202852 (+ 1997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:26:10.202870 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:26:10.203771 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:26:10.203778 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:26:10.204076 (+ 298us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:10.204871 (+ 795us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:26:10.204901 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:26:10.207853 (+ 2952us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:26:10.211775 (+ 3922us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:26:10.211783 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:26:10.211800 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:26:10.212295 (+ 495us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:26:10.212773 (+ 478us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:26:10.212777 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:26:10.212779 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:26:10.213007 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:26:10.213653 (+ 646us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:26:10.213661 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:26:10.214105 (+ 444us) client_negotiation.cc:769] Sending connection context
0810 03:26:10.214368 (+ 263us) client_negotiation.cc:240] Negotiation successful
0810 03:26:10.214844 (+ 476us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":509,"thread_start_us":266,"threads_started":1}
I20260810 03:26:10.217577 983 heartbeater.cc:347] Connected to a master server at 127.27.161.62:36693
I20260810 03:26:10.217936 983 heartbeater.cc:464] Registering TS with master...
I20260810 03:26:10.219132 983 heartbeater.cc:511] Master 127.27.161.62:36693 requested a full tablet report, sending...
I20260810 03:26:10.221069 510 ts_manager.cc:194] Registered new tserver with Master: 0165a059953946f6aabc53cb2aa727e2 (127.27.161.2:37641)
I20260810 03:26:10.221980 510 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.161.2@KRBTEST.COM'} at 127.27.161.2:35883
WARNING: no policy specified for kudu/127.27.161.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.161.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.161.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.3.keytab.
Entry for principal kudu/127.27.161.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.3.keytab.
I20260810 03:26:10.255419 28292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iBH5/build/asan/bin/kudu
/tmp/dist-test-task24iBH5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786332336319544-28292-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786332336319544-28292-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786332336319544-28292-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786332336319544-28292-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.161.3:0
--local_ip_for_outbound_sockets=127.27.161.3
--webserver_interface=127.27.161.3
--webserver_port=0
--tserver_master_addrs=127.27.161.62:36693
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.3.keytab
--principal=kudu/127.27.161.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.161.20:42417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786332336319544-28292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786332336319544-28292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786332336319544-28292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:26:10.601552 993 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:26:10.601866 993 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:26:10.601989 993 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:26:10.609401 993 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:26:10.609526 993 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:26:10.609642 993 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.161.3
I20260810 03:26:10.619946 993 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.161.20:42417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786332336319544-28292-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786332336319544-28292-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.3.keytab
--principal=kudu/127.27.161.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.161.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786332336319544-28292-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.161.3
--webserver_port=0
--tserver_master_addrs=127.27.161.62:36693
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.161.3
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786332336319544-28292-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:26:10.621300 993 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:26:10.622694 993 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:26:10.636132 1005 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:26:10.636210 1007 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:26:10.636161 1009 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:26:10.637346 993 server_base.cc:1034] running on GCE node
I20260810 03:26:10.638052 993 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:26:10.639014 993 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:26:10.640235 993 hybrid_clock.cc:648] HybridClock initialized: now 1786332370640178 us; error 48 us; skew 500 ppm
I20260810 03:26:10.640563 993 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:26:10 dist-test-slave-vpvm krb5kdc[447](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332370, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.161.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:26:10.643849 993 init.cc:377] Logged in from keytab as kudu/127.27.161.3@KRBTEST.COM (short username kudu)
I20260810 03:26:10.645434 993 webserver.cc:460] Webserver started at http://127.27.161.3:33851/ using document root <none> and password file <none>
I20260810 03:26:10.646198 993 fs_manager.cc:359] Metadata directory not provided
I20260810 03:26:10.646308 993 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:26:10.646596 993 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:26:10.650832 993 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786332336319544-28292-0/minicluster-data/ts-2/data/instance:
uuid: "945da9fa11114f3a88198f13f5528367"
format_stamp: "Formatted at 2026-08-10 03:26:10 on dist-test-slave-vpvm"
server_key: "152eb78400fd0a7b1a0908ecf51a31c0"
server_key_iv: "355f9c32c737ab2fc4a3a2b2cdbb092d"
server_key_version: "encryptionkey@0"
I20260810 03:26:10.652134 993 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786332336319544-28292-0/minicluster-data/ts-2/wal/instance:
uuid: "945da9fa11114f3a88198f13f5528367"
format_stamp: "Formatted at 2026-08-10 03:26:10 on dist-test-slave-vpvm"
server_key: "152eb78400fd0a7b1a0908ecf51a31c0"
server_key_iv: "355f9c32c737ab2fc4a3a2b2cdbb092d"
server_key_version: "encryptionkey@0"
I20260810 03:26:10.659284 993 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.007s
I20260810 03:26:10.663161 1018 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:26:10.664687 993 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260810 03:26:10.664892 993 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786332336319544-28292-0/minicluster-data/ts-2/data,/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786332336319544-28292-0/minicluster-data/ts-2/wal
uuid: "945da9fa11114f3a88198f13f5528367"
format_stamp: "Formatted at 2026-08-10 03:26:10 on dist-test-slave-vpvm"
server_key: "152eb78400fd0a7b1a0908ecf51a31c0"
server_key_iv: "355f9c32c737ab2fc4a3a2b2cdbb092d"
server_key_version: "encryptionkey@0"
I20260810 03:26:10.665102 993 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786332336319544-28292-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786332336319544-28292-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786332336319544-28292-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:26:10.684573 777 heartbeater.cc:503] Master 127.27.161.62:36693 was elected leader, sending a full tablet report...
I20260810 03:26:10.690267 993 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:26:10.694761 993 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:26:10.695075 993 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:26:10.696246 993 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:26:10.697700 993 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:26:10.697785 993 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:26:10.697912 993 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:26:10.697990 993 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260810 03:26:10.732692 993 rpc_server.cc:307] RPC server started. Bound to: 127.27.161.3:41471
I20260810 03:26:10.732759 1182 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.161.3:41471 every 8 connection(s)
I20260810 03:26:10.734398 993 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786332336319544-28292-0/minicluster-data/ts-2/data/info.pb
I20260810 03:26:10.741073 28292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iBH5/build/asan/bin/kudu as pid 993
I20260810 03:26:10.741256 28292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786332336319544-28292-0/minicluster-data/ts-2/wal/instance
I20260810 03:26:10.741705 28292 external_mini_cluster.cc:1428] Setting key 3f049dae2ad72051302322c6df301bea
Aug 10 03:26:10 dist-test-slave-vpvm krb5kdc[447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332370, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.161.3@KRBTEST.COM for kudu/127.27.161.62@KRBTEST.COM
I20260810 03:26:10.756673 1187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:10.737367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.161.3:58527 (local address 127.27.161.62:36693)
0810 03:26:10.737878 (+ 511us) server_negotiation.cc:206] Beginning negotiation
0810 03:26:10.737883 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:26:10.738834 (+ 951us) server_negotiation.cc:407] Connection header received
0810 03:26:10.740763 (+ 1929us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:26:10.740769 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:26:10.740873 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:26:10.740973 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:26:10.742893 (+ 1920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:10.744115 (+ 1222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:10.745778 (+ 1663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:10.746255 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:10.746371 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:10.750354 (+ 3983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:26:10.750361 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:26:10.750364 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:26:10.750404 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:26:10.752961 (+ 2557us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:26:10.753811 (+ 850us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:26:10.753816 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:26:10.753819 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:26:10.753985 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:26:10.754473 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:26:10.754477 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:26:10.754479 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:26:10.754795 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:26:10.754955 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0810 03:26:10.756009 (+ 1054us) server_negotiation.cc:299] Negotiation successful
0810 03:26:10.756286 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":372,"thread_start_us":236,"threads_started":1}
I20260810 03:26:10.757830 1188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:10.737881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:36693 (local address 127.27.161.3:58527)
0810 03:26:10.738667 (+ 786us) negotiation.cc:107] Waiting for socket to connect
0810 03:26:10.738706 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0810 03:26:10.740455 (+ 1749us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:26:10.741270 (+ 815us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:26:10.741283 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:26:10.741910 (+ 627us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:26:10.742706 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:26:10.742730 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:26:10.744306 (+ 1576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:26:10.744311 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:26:10.745620 (+ 1309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:26:10.745634 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:26:10.746511 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:26:10.746532 (+ 21us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:26:10.746765 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:10.747500 (+ 735us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:26:10.747545 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:26:10.750078 (+ 2533us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:26:10.753145 (+ 3067us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:26:10.753158 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:26:10.753174 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:26:10.753668 (+ 494us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:26:10.754109 (+ 441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:26:10.754114 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:26:10.754118 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:26:10.754343 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:26:10.754955 (+ 612us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:26:10.754964 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:26:10.755362 (+ 398us) client_negotiation.cc:769] Sending connection context
0810 03:26:10.755627 (+ 265us) client_negotiation.cc:240] Negotiation successful
0810 03:26:10.756415 (+ 788us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":534,"thread_start_us":277,"threads_started":1}
I20260810 03:26:10.759390 1183 heartbeater.cc:347] Connected to a master server at 127.27.161.62:36693
I20260810 03:26:10.759702 1183 heartbeater.cc:464] Registering TS with master...
I20260810 03:26:10.760480 1183 heartbeater.cc:511] Master 127.27.161.62:36693 requested a full tablet report, sending...
I20260810 03:26:10.762121 510 ts_manager.cc:194] Registered new tserver with Master: 945da9fa11114f3a88198f13f5528367 (127.27.161.3:41471)
I20260810 03:26:10.762992 510 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.161.3@KRBTEST.COM'} at 127.27.161.3:58527
I20260810 03:26:10.771174 28292 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 10 03:26:10 dist-test-slave-vpvm krb5kdc[447](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332370, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Aug 10 03:26:10 dist-test-slave-vpvm krb5kdc[447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332370, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.27.161.62@KRBTEST.COM
I20260810 03:26:10.831952 1187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:10.816683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39356 (local address 127.27.161.62:36693)
0810 03:26:10.816864 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0810 03:26:10.816869 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:26:10.817330 (+ 461us) server_negotiation.cc:407] Connection header received
0810 03:26:10.817519 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:26:10.817525 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:26:10.817601 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:26:10.817702 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:26:10.819235 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:10.820002 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:10.821277 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:10.821706 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:10.821843 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:10.825875 (+ 4032us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:26:10.825880 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:26:10.825884 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:26:10.825927 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:26:10.828784 (+ 2857us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:26:10.829597 (+ 813us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:26:10.829602 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:26:10.829605 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:26:10.829770 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:26:10.830190 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:26:10.830196 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:26:10.830199 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:26:10.830851 (+ 652us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:26:10.831129 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0810 03:26:10.831462 (+ 333us) server_negotiation.cc:299] Negotiation successful
0810 03:26:10.831699 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
W20260810 03:26:10.832579 510 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:39356
I20260810 03:26:10.834409 28292 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iBH5/build/asan/bin/kudu pid 597
Aug 10 03:26:10 dist-test-slave-vpvm krb5kdc[447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332370, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.27.161.1@KRBTEST.COM
I20260810 03:26:10.852861 1203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:10.835783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34188 (local address 127.27.161.1:37295)
0810 03:26:10.836440 (+ 657us) server_negotiation.cc:206] Beginning negotiation
0810 03:26:10.836447 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:26:10.836474 (+ 27us) server_negotiation.cc:407] Connection header received
0810 03:26:10.836534 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:26:10.836541 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:26:10.836758 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:26:10.836886 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:26:10.838392 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:10.839629 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:10.840922 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:10.841550 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:10.841675 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:10.845921 (+ 4246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:26:10.845939 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:26:10.846008 (+ 69us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:26:10.846057 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:26:10.849357 (+ 3300us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:26:10.850777 (+ 1420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:26:10.850786 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:26:10.850789 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:26:10.850955 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:26:10.851335 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:26:10.851339 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:26:10.851341 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:26:10.851743 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:26:10.851990 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0810 03:26:10.852336 (+ 346us) server_negotiation.cc:299] Negotiation successful
0810 03:26:10.852554 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":495,"thread_start_us":331,"threads_started":1}
W20260810 03:26:10.854316 771 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:34188
W20260810 03:26:10.855492 28292 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task24iBH5/build/asan/bin/kudu pid 597: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260810 03:26:10.855724 28292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iBH5/build/asan/bin/kudu with pid 597
I20260810 03:26:10.884260 28292 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iBH5/build/asan/bin/kudu pid 788
Aug 10 03:26:10 dist-test-slave-vpvm krb5kdc[447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332370, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.27.161.2@KRBTEST.COM
I20260810 03:26:10.901000 1204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:10.885291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58388 (local address 127.27.161.2:37641)
0810 03:26:10.885906 (+ 615us) server_negotiation.cc:206] Beginning negotiation
0810 03:26:10.885914 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 03:26:10.885940 (+ 26us) server_negotiation.cc:407] Connection header received
0810 03:26:10.886051 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:26:10.886058 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:26:10.886282 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:26:10.886425 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:26:10.887649 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:10.888755 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:10.890049 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:10.890659 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:10.890775 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:10.894989 (+ 4214us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:26:10.895006 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:26:10.895024 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:26:10.895067 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:26:10.897862 (+ 2795us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:26:10.898752 (+ 890us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:26:10.898761 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:26:10.898763 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:26:10.898900 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:26:10.899277 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:26:10.899281 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:26:10.899283 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:26:10.899687 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:26:10.900007 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0810 03:26:10.900368 (+ 361us) server_negotiation.cc:299] Negotiation successful
0810 03:26:10.900661 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":475,"thread_start_us":368,"threads_started":1}
W20260810 03:26:10.902387 977 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:58388
W20260810 03:26:10.903637 28292 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task24iBH5/build/asan/bin/kudu pid 788: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260810 03:26:10.903786 28292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iBH5/build/asan/bin/kudu with pid 788
I20260810 03:26:10.927534 28292 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iBH5/build/asan/bin/kudu pid 993
Aug 10 03:26:10 dist-test-slave-vpvm krb5kdc[447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332370, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.27.161.3@KRBTEST.COM
I20260810 03:26:10.944921 1205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:10.928653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41708 (local address 127.27.161.3:41471)
0810 03:26:10.929311 (+ 658us) server_negotiation.cc:206] Beginning negotiation
0810 03:26:10.929318 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:26:10.929348 (+ 30us) server_negotiation.cc:407] Connection header received
0810 03:26:10.929411 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:26:10.929418 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:26:10.929666 (+ 248us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:26:10.929824 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:26:10.931220 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:10.932279 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:10.933540 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:10.934276 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:10.934396 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:10.938854 (+ 4458us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:26:10.938870 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:26:10.938887 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:26:10.938933 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:26:10.941688 (+ 2755us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:26:10.942628 (+ 940us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:26:10.942636 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:26:10.942638 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:26:10.942777 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:26:10.943149 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:26:10.943153 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:26:10.943155 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:26:10.943612 (+ 457us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:26:10.943918 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0810 03:26:10.944259 (+ 341us) server_negotiation.cc:299] Negotiation successful
0810 03:26:10.944551 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":509,"thread_start_us":353,"threads_started":1}
W20260810 03:26:10.946298 1177 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:41708
W20260810 03:26:10.947422 28292 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task24iBH5/build/asan/bin/kudu pid 993: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260810 03:26:10.947594 28292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iBH5/build/asan/bin/kudu with pid 993
I20260810 03:26:10.978004 28292 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iBH5/build/asan/bin/kudu pid 465
I20260810 03:26:10.979281 554 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:26:11.085472 28292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iBH5/build/asan/bin/kudu with pid 465
2026-08-10T03:26:11Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (4011 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786332336319544-28292-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 03:26:11 dist-test-slave-vpvm krb5kdc[1209](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 03:26:11 dist-test-slave-vpvm krb5kdc[1209](info): set up 2 sockets
Aug 10 03:26:11 dist-test-slave-vpvm krb5kdc[1209](info): commencing operation
krb5kdc: starting...
W20260810 03:26:12.319813 28292 mini_kdc.cc:121] Time spent starting KDC: real 1.183s user 0.000s sys 0.032s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786332336319544-28292-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786332336319544-28292-0/krb5kdc/test-user.keytab.
Aug 10 03:26:12 dist-test-slave-vpvm krb5kdc[1209](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332372, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T03:26:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T03:26:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.161.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.161.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.161.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.62.keytab.
Entry for principal kudu/127.27.161.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.62.keytab.
I20260810 03:26:12.565570 28292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iBH5/build/asan/bin/kudu
/tmp/dist-test-task24iBH5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786332336319544-28292-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786332336319544-28292-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786332336319544-28292-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786332336319544-28292-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.161.62:36881
--webserver_interface=127.27.161.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.62.keytab
--principal=kudu/127.27.161.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.161.20:46307
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.161.62:36881
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786332336319544-28292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786332336319544-28292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786332336319544-28292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:26:12.911856 1229 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:26:12.912187 1229 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:26:12.912321 1229 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:26:12.919731 1229 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:26:12.919853 1229 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:26:12.919937 1229 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:26:12.920027 1229 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:26:12.920083 1229 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:26:12.930480 1229 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.161.20:46307
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786332336319544-28292-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786332336319544-28292-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.161.62:36881
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.62.keytab
--principal=kudu/127.27.161.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.161.62:36881
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786332336319544-28292-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.161.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786332336319544-28292-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:26:12.931886 1229 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:26:12.933331 1229 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:26:12.945657 1245 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:26:12.945560 1239 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:26:12.945590 1242 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:26:12.946429 1229 server_base.cc:1034] running on GCE node
I20260810 03:26:12.947250 1229 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:26:12.948865 1229 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:26:12.950093 1229 hybrid_clock.cc:648] HybridClock initialized: now 1786332372950025 us; error 54 us; skew 500 ppm
I20260810 03:26:12.950507 1229 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:26:12 dist-test-slave-vpvm krb5kdc[1209](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332372, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.161.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:26:12.953847 1229 init.cc:377] Logged in from keytab as kudu/127.27.161.62@KRBTEST.COM (short username kudu)
I20260810 03:26:12.955482 1229 webserver.cc:460] Webserver started at http://127.27.161.62:45653/ using document root <none> and password file <none>
I20260810 03:26:12.956261 1229 fs_manager.cc:359] Metadata directory not provided
I20260810 03:26:12.956388 1229 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:26:12.956806 1229 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:26:12.960920 1229 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786332336319544-28292-0/minicluster-data/master-0/data/instance:
uuid: "20c148dad7a242959fb02411d3bcac14"
format_stamp: "Formatted at 2026-08-10 03:26:12 on dist-test-slave-vpvm"
server_key: "723cab6a251634a9dbb233bced1dd84c"
server_key_iv: "0580a6631e73847aa0aa2064b7720f95"
server_key_version: "encryptionkey@0"
I20260810 03:26:12.962319 1229 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786332336319544-28292-0/minicluster-data/master-0/wal/instance:
uuid: "20c148dad7a242959fb02411d3bcac14"
format_stamp: "Formatted at 2026-08-10 03:26:12 on dist-test-slave-vpvm"
server_key: "723cab6a251634a9dbb233bced1dd84c"
server_key_iv: "0580a6631e73847aa0aa2064b7720f95"
server_key_version: "encryptionkey@0"
I20260810 03:26:12.968546 1229 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260810 03:26:12.972535 1257 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:26:12.974319 1229 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 03:26:12.974716 1229 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786332336319544-28292-0/minicluster-data/master-0/data,/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786332336319544-28292-0/minicluster-data/master-0/wal
uuid: "20c148dad7a242959fb02411d3bcac14"
format_stamp: "Formatted at 2026-08-10 03:26:12 on dist-test-slave-vpvm"
server_key: "723cab6a251634a9dbb233bced1dd84c"
server_key_iv: "0580a6631e73847aa0aa2064b7720f95"
server_key_version: "encryptionkey@0"
I20260810 03:26:12.975083 1229 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786332336319544-28292-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786332336319544-28292-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786332336319544-28292-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:26:12.990249 1229 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:26:12.994974 1229 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:26:12.995268 1229 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:26:13.013630 1229 rpc_server.cc:307] RPC server started. Bound to: 127.27.161.62:36881
I20260810 03:26:13.013687 1328 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.161.62:36881 every 8 connection(s)
I20260810 03:26:13.015364 1229 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786332336319544-28292-0/minicluster-data/master-0/data/info.pb
I20260810 03:26:13.019853 1329 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:26:13.021582 28292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iBH5/build/asan/bin/kudu as pid 1229
I20260810 03:26:13.021840 28292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786332336319544-28292-0/minicluster-data/master-0/wal/instance
I20260810 03:26:13.022684 28292 external_mini_cluster.cc:1428] Setting key 581681400f3c1e83f1981996c737f266
I20260810 03:26:13.033545 1329 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 20c148dad7a242959fb02411d3bcac14: Bootstrap starting.
I20260810 03:26:13.037761 1329 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 20c148dad7a242959fb02411d3bcac14: Neither blocks nor log segments found. Creating new log.
Aug 10 03:26:13 dist-test-slave-vpvm krb5kdc[1209](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332372, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.161.62@KRBTEST.COM
I20260810 03:26:13.039072 1329 log.cc:824] T 00000000000000000000000000000000 P 20c148dad7a242959fb02411d3bcac14: Log is configured to *not* fsync() on all Append() calls
I20260810 03:26:13.042837 1329 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 20c148dad7a242959fb02411d3bcac14: No bootstrap required, opened a new log
I20260810 03:26:13.049220 1335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:13.025111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33340 (local address 127.27.161.62:36881)
0810 03:26:13.026106 (+ 995us) server_negotiation.cc:206] Beginning negotiation
0810 03:26:13.026120 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 03:26:13.026176 (+ 56us) server_negotiation.cc:407] Connection header received
0810 03:26:13.028095 (+ 1919us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:26:13.028122 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:26:13.028560 (+ 438us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:26:13.028923 (+ 363us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:26:13.030878 (+ 1955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:13.032349 (+ 1471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:13.034135 (+ 1786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:13.034903 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:13.035071 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:13.040223 (+ 5152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:26:13.040237 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:26:13.040260 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:26:13.040319 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:26:13.044548 (+ 4229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:26:13.045501 (+ 953us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:26:13.045508 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:26:13.045519 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:26:13.045736 (+ 217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:26:13.046202 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:26:13.046206 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:26:13.046207 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:26:13.046866 (+ 659us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:26:13.047116 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0810 03:26:13.047591 (+ 475us) server_negotiation.cc:299] Negotiation successful
0810 03:26:13.048056 (+ 465us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":674,"thread_start_us":421,"threads_started":1}
I20260810 03:26:13.051916 1329 raft_consensus.cc:348] T 00000000000000000000000000000000 P 20c148dad7a242959fb02411d3bcac14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20c148dad7a242959fb02411d3bcac14" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 36881 } }
I20260810 03:26:13.052309 1329 raft_consensus.cc:374] T 00000000000000000000000000000000 P 20c148dad7a242959fb02411d3bcac14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:26:13.052407 1329 raft_consensus.cc:729] T 00000000000000000000000000000000 P 20c148dad7a242959fb02411d3bcac14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20c148dad7a242959fb02411d3bcac14, State: Initialized, Role: FOLLOWER
I20260810 03:26:13.052989 1329 consensus_queue.cc:260] T 00000000000000000000000000000000 P 20c148dad7a242959fb02411d3bcac14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20c148dad7a242959fb02411d3bcac14" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 36881 } }
I20260810 03:26:13.053227 1329 raft_consensus.cc:388] T 00000000000000000000000000000000 P 20c148dad7a242959fb02411d3bcac14 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 03:26:13.053354 1329 raft_consensus.cc:482] T 00000000000000000000000000000000 P 20c148dad7a242959fb02411d3bcac14 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 03:26:13.053511 1329 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 20c148dad7a242959fb02411d3bcac14 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:26:13.055876 1329 raft_consensus.cc:504] T 00000000000000000000000000000000 P 20c148dad7a242959fb02411d3bcac14 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20c148dad7a242959fb02411d3bcac14" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 36881 } }
I20260810 03:26:13.056473 1329 leader_election.cc:302] T 00000000000000000000000000000000 P 20c148dad7a242959fb02411d3bcac14 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 20c148dad7a242959fb02411d3bcac14; no voters:
I20260810 03:26:13.057126 1329 leader_election.cc:288] T 00000000000000000000000000000000 P 20c148dad7a242959fb02411d3bcac14 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 03:26:13.057294 1337 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 20c148dad7a242959fb02411d3bcac14 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:26:13.057983 1337 raft_consensus.cc:686] T 00000000000000000000000000000000 P 20c148dad7a242959fb02411d3bcac14 [term 1 LEADER]: Becoming Leader. State: Replica: 20c148dad7a242959fb02411d3bcac14, State: Running, Role: LEADER
I20260810 03:26:13.058554 1337 consensus_queue.cc:237] T 00000000000000000000000000000000 P 20c148dad7a242959fb02411d3bcac14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20c148dad7a242959fb02411d3bcac14" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 36881 } }
I20260810 03:26:13.059345 1329 sys_catalog.cc:564] T 00000000000000000000000000000000 P 20c148dad7a242959fb02411d3bcac14 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:26:13.061213 1339 sys_catalog.cc:455] T 00000000000000000000000000000000 P 20c148dad7a242959fb02411d3bcac14 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 20c148dad7a242959fb02411d3bcac14. Latest consensus state: current_term: 1 leader_uuid: "20c148dad7a242959fb02411d3bcac14" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20c148dad7a242959fb02411d3bcac14" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 36881 } } }
I20260810 03:26:13.062045 1339 sys_catalog.cc:458] T 00000000000000000000000000000000 P 20c148dad7a242959fb02411d3bcac14 [sys.catalog]: This master's current role is: LEADER
I20260810 03:26:13.061389 1338 sys_catalog.cc:455] T 00000000000000000000000000000000 P 20c148dad7a242959fb02411d3bcac14 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "20c148dad7a242959fb02411d3bcac14" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20c148dad7a242959fb02411d3bcac14" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 36881 } } }
I20260810 03:26:13.062597 1338 sys_catalog.cc:458] T 00000000000000000000000000000000 P 20c148dad7a242959fb02411d3bcac14 [sys.catalog]: This master's current role is: LEADER
I20260810 03:26:13.067333 1342 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 03:26:13.073004 1342 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 03:26:13.084363 1342 catalog_manager.cc:1347] Generated new cluster ID: 9781db30d3c44a9aabf4655d8b34d04d
I20260810 03:26:13.084501 1342 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 03:26:13.106700 1342 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 03:26:13.108127 1342 catalog_manager.cc:1504] Loading token signing keys...
I20260810 03:26:13.118314 1342 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 20c148dad7a242959fb02411d3bcac14: Generated new TSK 0
I20260810 03:26:13.119277 1342 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.161.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.161.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.161.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.1.keytab.
Entry for principal kudu/127.27.161.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.1.keytab.
I20260810 03:26:13.191757 28292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iBH5/build/asan/bin/kudu
/tmp/dist-test-task24iBH5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786332336319544-28292-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786332336319544-28292-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786332336319544-28292-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786332336319544-28292-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.161.1:0
--local_ip_for_outbound_sockets=127.27.161.1
--webserver_interface=127.27.161.1
--webserver_port=0
--tserver_master_addrs=127.27.161.62:36881
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.1.keytab
--principal=kudu/127.27.161.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.161.20:46307
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786332336319544-28292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786332336319544-28292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786332336319544-28292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:26:13.568051 1361 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:26:13.568375 1361 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:26:13.568615 1361 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:26:13.576630 1361 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:26:13.576761 1361 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:26:13.576917 1361 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.161.1
I20260810 03:26:13.587476 1361 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.161.20:46307
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786332336319544-28292-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786332336319544-28292-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.1.keytab
--principal=kudu/127.27.161.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.161.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786332336319544-28292-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.161.1
--webserver_port=0
--tserver_master_addrs=127.27.161.62:36881
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.161.1
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786332336319544-28292-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:26:13.588845 1361 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:26:13.590317 1361 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:26:13.604305 1366 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:26:13.604216 1372 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:26:13.604264 1368 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:26:13.605276 1361 server_base.cc:1034] running on GCE node
I20260810 03:26:13.605930 1361 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:26:13.607054 1361 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:26:13.608307 1361 hybrid_clock.cc:648] HybridClock initialized: now 1786332373608251 us; error 43 us; skew 500 ppm
I20260810 03:26:13.608700 1361 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:26:13 dist-test-slave-vpvm krb5kdc[1209](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332373, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.161.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:26:13.612358 1361 init.cc:377] Logged in from keytab as kudu/127.27.161.1@KRBTEST.COM (short username kudu)
I20260810 03:26:13.614045 1361 webserver.cc:460] Webserver started at http://127.27.161.1:38387/ using document root <none> and password file <none>
I20260810 03:26:13.614786 1361 fs_manager.cc:359] Metadata directory not provided
I20260810 03:26:13.614918 1361 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:26:13.615265 1361 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:26:13.618468 1361 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786332336319544-28292-0/minicluster-data/ts-0/data/instance:
uuid: "05298817f34b459798eed009b9b8cbc0"
format_stamp: "Formatted at 2026-08-10 03:26:13 on dist-test-slave-vpvm"
server_key: "f5693d8ec242389bb564bbe3c5fb20f0"
server_key_iv: "ae1a7c7d36b76a83fcf03f02b1095e86"
server_key_version: "encryptionkey@0"
I20260810 03:26:13.619298 1361 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786332336319544-28292-0/minicluster-data/ts-0/wal/instance:
uuid: "05298817f34b459798eed009b9b8cbc0"
format_stamp: "Formatted at 2026-08-10 03:26:13 on dist-test-slave-vpvm"
server_key: "f5693d8ec242389bb564bbe3c5fb20f0"
server_key_iv: "ae1a7c7d36b76a83fcf03f02b1095e86"
server_key_version: "encryptionkey@0"
I20260810 03:26:13.625061 1361 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260810 03:26:13.628830 1385 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:26:13.630321 1361 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 03:26:13.630509 1361 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786332336319544-28292-0/minicluster-data/ts-0/data,/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786332336319544-28292-0/minicluster-data/ts-0/wal
uuid: "05298817f34b459798eed009b9b8cbc0"
format_stamp: "Formatted at 2026-08-10 03:26:13 on dist-test-slave-vpvm"
server_key: "f5693d8ec242389bb564bbe3c5fb20f0"
server_key_iv: "ae1a7c7d36b76a83fcf03f02b1095e86"
server_key_version: "encryptionkey@0"
I20260810 03:26:13.630722 1361 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786332336319544-28292-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786332336319544-28292-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786332336319544-28292-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:26:13.643877 1361 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:26:13.648474 1361 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:26:13.648801 1361 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:26:13.649895 1361 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:26:13.651695 1361 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:26:13.651789 1361 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:26:13.651916 1361 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:26:13.651975 1361 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:26:13.686544 1361 rpc_server.cc:307] RPC server started. Bound to: 127.27.161.1:41029
I20260810 03:26:13.686617 1543 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.161.1:41029 every 8 connection(s)
I20260810 03:26:13.687909 1361 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786332336319544-28292-0/minicluster-data/ts-0/data/info.pb
I20260810 03:26:13.693204 28292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iBH5/build/asan/bin/kudu as pid 1361
I20260810 03:26:13.693403 28292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786332336319544-28292-0/minicluster-data/ts-0/wal/instance
I20260810 03:26:13.694094 28292 external_mini_cluster.cc:1428] Setting key df4317a4e86812b19f4e91c9efd10ada
Aug 10 03:26:13 dist-test-slave-vpvm krb5kdc[1209](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332373, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.161.1@KRBTEST.COM for kudu/127.27.161.62@KRBTEST.COM
I20260810 03:26:13.709746 1548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:13.690832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.161.1:39039 (local address 127.27.161.62:36881)
0810 03:26:13.691292 (+ 460us) server_negotiation.cc:206] Beginning negotiation
0810 03:26:13.691297 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:26:13.692390 (+ 1093us) server_negotiation.cc:407] Connection header received
0810 03:26:13.694491 (+ 2101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:26:13.694497 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:26:13.694603 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:26:13.694734 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:26:13.696878 (+ 2144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:13.697854 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:13.699486 (+ 1632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:13.700027 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:13.700149 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:13.703806 (+ 3657us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:26:13.703813 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:26:13.703817 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:26:13.703857 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:26:13.706607 (+ 2750us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:26:13.707481 (+ 874us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:26:13.707485 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:26:13.707487 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:26:13.707618 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:26:13.708087 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:26:13.708090 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:26:13.708092 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:26:13.708352 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:26:13.708461 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0810 03:26:13.709300 (+ 839us) server_negotiation.cc:299] Negotiation successful
0810 03:26:13.709471 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334,"thread_start_us":218,"threads_started":1}
I20260810 03:26:13.711257 1549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:13.691430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:36881 (local address 127.27.161.1:39039)
0810 03:26:13.692209 (+ 779us) negotiation.cc:107] Waiting for socket to connect
0810 03:26:13.692259 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0810 03:26:13.694181 (+ 1922us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:26:13.695092 (+ 911us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:26:13.695102 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:26:13.695876 (+ 774us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:26:13.696682 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:26:13.696703 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:26:13.698051 (+ 1348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:26:13.698055 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:26:13.699332 (+ 1277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:26:13.699346 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:26:13.700252 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:26:13.700257 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:26:13.700469 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:13.701155 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:26:13.701179 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:26:13.703555 (+ 2376us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:26:13.706760 (+ 3205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:26:13.706770 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:26:13.706791 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:26:13.707381 (+ 590us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:26:13.707718 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:26:13.707723 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:26:13.707727 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:26:13.707972 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:26:13.708473 (+ 501us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:26:13.708481 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:26:13.708914 (+ 433us) client_negotiation.cc:769] Sending connection context
0810 03:26:13.709229 (+ 315us) client_negotiation.cc:240] Negotiation successful
0810 03:26:13.709846 (+ 617us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":531,"thread_start_us":298,"threads_started":1}
I20260810 03:26:13.712857 1544 heartbeater.cc:347] Connected to a master server at 127.27.161.62:36881
I20260810 03:26:13.713165 1544 heartbeater.cc:464] Registering TS with master...
I20260810 03:26:13.713917 1544 heartbeater.cc:511] Master 127.27.161.62:36881 requested a full tablet report, sending...
I20260810 03:26:13.716361 1279 ts_manager.cc:194] Registered new tserver with Master: 05298817f34b459798eed009b9b8cbc0 (127.27.161.1:41029)
I20260810 03:26:13.718575 1279 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.161.1@KRBTEST.COM'} at 127.27.161.1:39039
WARNING: no policy specified for kudu/127.27.161.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.161.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.161.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.2.keytab.
Entry for principal kudu/127.27.161.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.2.keytab.
I20260810 03:26:13.759084 28292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iBH5/build/asan/bin/kudu
/tmp/dist-test-task24iBH5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786332336319544-28292-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786332336319544-28292-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786332336319544-28292-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786332336319544-28292-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.161.2:0
--local_ip_for_outbound_sockets=127.27.161.2
--webserver_interface=127.27.161.2
--webserver_port=0
--tserver_master_addrs=127.27.161.62:36881
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.2.keytab
--principal=kudu/127.27.161.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.161.20:46307
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786332336319544-28292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786332336319544-28292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786332336319544-28292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:26:14.110988 1553 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:26:14.111292 1553 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:26:14.111413 1553 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:26:14.119041 1553 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:26:14.119168 1553 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:26:14.119316 1553 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.161.2
I20260810 03:26:14.129549 1553 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.161.20:46307
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786332336319544-28292-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786332336319544-28292-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.2.keytab
--principal=kudu/127.27.161.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.161.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786332336319544-28292-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.161.2
--webserver_port=0
--tserver_master_addrs=127.27.161.62:36881
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.161.2
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786332336319544-28292-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:26:14.131016 1553 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:26:14.132388 1553 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:26:14.145888 1560 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:26:14.146078 1565 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:26:14.145869 1562 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:26:14.146921 1553 server_base.cc:1034] running on GCE node
I20260810 03:26:14.147622 1553 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:26:14.148680 1553 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:26:14.149901 1553 hybrid_clock.cc:648] HybridClock initialized: now 1786332374149844 us; error 48 us; skew 500 ppm
I20260810 03:26:14.150274 1553 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:26:14 dist-test-slave-vpvm krb5kdc[1209](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332374, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.161.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:26:14.153517 1553 init.cc:377] Logged in from keytab as kudu/127.27.161.2@KRBTEST.COM (short username kudu)
I20260810 03:26:14.155236 1553 webserver.cc:460] Webserver started at http://127.27.161.2:41643/ using document root <none> and password file <none>
I20260810 03:26:14.156163 1553 fs_manager.cc:359] Metadata directory not provided
I20260810 03:26:14.156275 1553 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:26:14.156558 1553 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:26:14.159718 1553 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786332336319544-28292-0/minicluster-data/ts-1/data/instance:
uuid: "ffb07fd0e58849dfacc8b687aaf6bb94"
format_stamp: "Formatted at 2026-08-10 03:26:14 on dist-test-slave-vpvm"
server_key: "f5a64712e139442d6b990da19eb02760"
server_key_iv: "87e5e12b1354eddfabbd6a358979c0d5"
server_key_version: "encryptionkey@0"
I20260810 03:26:14.160562 1553 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786332336319544-28292-0/minicluster-data/ts-1/wal/instance:
uuid: "ffb07fd0e58849dfacc8b687aaf6bb94"
format_stamp: "Formatted at 2026-08-10 03:26:14 on dist-test-slave-vpvm"
server_key: "f5a64712e139442d6b990da19eb02760"
server_key_iv: "87e5e12b1354eddfabbd6a358979c0d5"
server_key_version: "encryptionkey@0"
I20260810 03:26:14.166232 1553 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260810 03:26:14.170135 1577 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:26:14.171773 1553 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 03:26:14.171964 1553 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786332336319544-28292-0/minicluster-data/ts-1/data,/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786332336319544-28292-0/minicluster-data/ts-1/wal
uuid: "ffb07fd0e58849dfacc8b687aaf6bb94"
format_stamp: "Formatted at 2026-08-10 03:26:14 on dist-test-slave-vpvm"
server_key: "f5a64712e139442d6b990da19eb02760"
server_key_iv: "87e5e12b1354eddfabbd6a358979c0d5"
server_key_version: "encryptionkey@0"
I20260810 03:26:14.172197 1553 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786332336319544-28292-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786332336319544-28292-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786332336319544-28292-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:26:14.192293 1553 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:26:14.196661 1553 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:26:14.196954 1553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:26:14.198026 1553 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:26:14.199672 1553 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:26:14.199755 1553 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:26:14.199901 1553 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:26:14.199955 1553 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:26:14.238788 1553 rpc_server.cc:307] RPC server started. Bound to: 127.27.161.2:45459
I20260810 03:26:14.238845 1746 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.161.2:45459 every 8 connection(s)
I20260810 03:26:14.240245 1553 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786332336319544-28292-0/minicluster-data/ts-1/data/info.pb
I20260810 03:26:14.246351 28292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iBH5/build/asan/bin/kudu as pid 1553
I20260810 03:26:14.246553 28292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786332336319544-28292-0/minicluster-data/ts-1/wal/instance
I20260810 03:26:14.246959 28292 external_mini_cluster.cc:1428] Setting key df8c6d38cb136e0741b3278bb49a0d4a
Aug 10 03:26:14 dist-test-slave-vpvm krb5kdc[1209](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332374, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.161.2@KRBTEST.COM for kudu/127.27.161.62@KRBTEST.COM
I20260810 03:26:14.265257 1754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:14.242979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.161.2:42937 (local address 127.27.161.62:36881)
0810 03:26:14.243485 (+ 506us) server_negotiation.cc:206] Beginning negotiation
0810 03:26:14.243495 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 03:26:14.244391 (+ 896us) server_negotiation.cc:407] Connection header received
0810 03:26:14.246280 (+ 1889us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:26:14.246286 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:26:14.246373 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:26:14.246472 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:26:14.248310 (+ 1838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:14.249507 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:14.251887 (+ 2380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:14.252539 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:14.252675 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:14.258346 (+ 5671us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:26:14.258353 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:26:14.258357 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:26:14.258397 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:26:14.261508 (+ 3111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:26:14.262449 (+ 941us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:26:14.262456 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:26:14.262460 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:26:14.262628 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:26:14.263161 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:26:14.263166 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:26:14.263169 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:26:14.263612 (+ 443us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:26:14.263773 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0810 03:26:14.264561 (+ 788us) server_negotiation.cc:299] Negotiation successful
0810 03:26:14.264845 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":381,"thread_start_us":273,"threads_started":1}
I20260810 03:26:14.266355 1755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:14.243455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:36881 (local address 127.27.161.2:42937)
0810 03:26:14.244207 (+ 752us) negotiation.cc:107] Waiting for socket to connect
0810 03:26:14.244251 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0810 03:26:14.246002 (+ 1751us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:26:14.246747 (+ 745us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:26:14.246757 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:26:14.247312 (+ 555us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:26:14.248050 (+ 738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:26:14.248077 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:26:14.249747 (+ 1670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:26:14.249753 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:26:14.251719 (+ 1966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:26:14.251738 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:26:14.252922 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:26:14.252930 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:26:14.253238 (+ 308us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:14.254948 (+ 1710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:26:14.255008 (+ 60us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:26:14.258016 (+ 3008us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:26:14.261691 (+ 3675us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:26:14.261700 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:26:14.261717 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:26:14.262296 (+ 579us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:26:14.262775 (+ 479us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:26:14.262779 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:26:14.262782 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:26:14.263011 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:26:14.263764 (+ 753us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:26:14.263772 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:26:14.264159 (+ 387us) client_negotiation.cc:769] Sending connection context
0810 03:26:14.264416 (+ 257us) client_negotiation.cc:240] Negotiation successful
0810 03:26:14.265026 (+ 610us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":276,"threads_started":1}
I20260810 03:26:14.267956 1747 heartbeater.cc:347] Connected to a master server at 127.27.161.62:36881
I20260810 03:26:14.268381 1747 heartbeater.cc:464] Registering TS with master...
I20260810 03:26:14.269359 1747 heartbeater.cc:511] Master 127.27.161.62:36881 requested a full tablet report, sending...
I20260810 03:26:14.271265 1279 ts_manager.cc:194] Registered new tserver with Master: ffb07fd0e58849dfacc8b687aaf6bb94 (127.27.161.2:45459)
I20260810 03:26:14.272145 1279 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.161.2@KRBTEST.COM'} at 127.27.161.2:42937
WARNING: no policy specified for kudu/127.27.161.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.161.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.161.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.3.keytab.
Entry for principal kudu/127.27.161.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.3.keytab.
I20260810 03:26:14.304790 28292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iBH5/build/asan/bin/kudu
/tmp/dist-test-task24iBH5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786332336319544-28292-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786332336319544-28292-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786332336319544-28292-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786332336319544-28292-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.161.3:0
--local_ip_for_outbound_sockets=127.27.161.3
--webserver_interface=127.27.161.3
--webserver_port=0
--tserver_master_addrs=127.27.161.62:36881
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.3.keytab
--principal=kudu/127.27.161.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.161.20:46307
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786332336319544-28292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786332336319544-28292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786332336319544-28292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:26:14.652051 1760 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:26:14.652369 1760 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:26:14.652500 1760 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:26:14.659961 1760 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:26:14.660117 1760 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:26:14.660279 1760 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.161.3
I20260810 03:26:14.670444 1760 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.161.20:46307
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786332336319544-28292-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786332336319544-28292-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.3.keytab
--principal=kudu/127.27.161.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.161.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786332336319544-28292-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.161.3
--webserver_port=0
--tserver_master_addrs=127.27.161.62:36881
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.161.3
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786332336319544-28292-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:26:14.671948 1760 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:26:14.673357 1760 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:26:14.688239 1770 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:26:14.688719 1767 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:26:14.688385 1766 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:26:14.689291 1760 server_base.cc:1034] running on GCE node
I20260810 03:26:14.689919 1760 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:26:14.691064 1760 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:26:14.692283 1760 hybrid_clock.cc:648] HybridClock initialized: now 1786332374692231 us; error 47 us; skew 500 ppm
I20260810 03:26:14.692581 1760 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:26:14 dist-test-slave-vpvm krb5kdc[1209](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332374, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.161.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:26:14.695839 1760 init.cc:377] Logged in from keytab as kudu/127.27.161.3@KRBTEST.COM (short username kudu)
I20260810 03:26:14.697376 1760 webserver.cc:460] Webserver started at http://127.27.161.3:42797/ using document root <none> and password file <none>
I20260810 03:26:14.698119 1760 fs_manager.cc:359] Metadata directory not provided
I20260810 03:26:14.698232 1760 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:26:14.698509 1760 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:26:14.701622 1760 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786332336319544-28292-0/minicluster-data/ts-2/data/instance:
uuid: "4c2e595f4c71400793c6588195eac760"
format_stamp: "Formatted at 2026-08-10 03:26:14 on dist-test-slave-vpvm"
server_key: "470f525409b9b4aff221642da8524079"
server_key_iv: "0cbd6bf484465e1674f11ff527e0dc84"
server_key_version: "encryptionkey@0"
I20260810 03:26:14.702549 1760 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786332336319544-28292-0/minicluster-data/ts-2/wal/instance:
uuid: "4c2e595f4c71400793c6588195eac760"
format_stamp: "Formatted at 2026-08-10 03:26:14 on dist-test-slave-vpvm"
server_key: "470f525409b9b4aff221642da8524079"
server_key_iv: "0cbd6bf484465e1674f11ff527e0dc84"
server_key_version: "encryptionkey@0"
I20260810 03:26:14.708320 1760 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260810 03:26:14.712141 1778 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:26:14.713567 1760 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 03:26:14.713752 1760 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786332336319544-28292-0/minicluster-data/ts-2/data,/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786332336319544-28292-0/minicluster-data/ts-2/wal
uuid: "4c2e595f4c71400793c6588195eac760"
format_stamp: "Formatted at 2026-08-10 03:26:14 on dist-test-slave-vpvm"
server_key: "470f525409b9b4aff221642da8524079"
server_key_iv: "0cbd6bf484465e1674f11ff527e0dc84"
server_key_version: "encryptionkey@0"
I20260810 03:26:14.714018 1760 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786332336319544-28292-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786332336319544-28292-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786332336319544-28292-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:26:14.721866 1544 heartbeater.cc:503] Master 127.27.161.62:36881 was elected leader, sending a full tablet report...
I20260810 03:26:14.734392 1760 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:26:14.738647 1760 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:26:14.738946 1760 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:26:14.739997 1760 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:26:14.741647 1760 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:26:14.741739 1760 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:26:14.741870 1760 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:26:14.741923 1760 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:26:14.776355 1760 rpc_server.cc:307] RPC server started. Bound to: 127.27.161.3:38839
I20260810 03:26:14.776467 1945 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.161.3:38839 every 8 connection(s)
I20260810 03:26:14.777846 1760 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786332336319544-28292-0/minicluster-data/ts-2/data/info.pb
I20260810 03:26:14.780627 28292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iBH5/build/asan/bin/kudu as pid 1760
I20260810 03:26:14.780833 28292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786332336319544-28292-0/minicluster-data/ts-2/wal/instance
I20260810 03:26:14.781370 28292 external_mini_cluster.cc:1428] Setting key 6d25787e23939e85d80b4e0782786a53
Aug 10 03:26:14 dist-test-slave-vpvm krb5kdc[1209](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332374, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.161.3@KRBTEST.COM for kudu/127.27.161.62@KRBTEST.COM
I20260810 03:26:14.801383 1948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:14.780778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.161.3:39561 (local address 127.27.161.62:36881)
0810 03:26:14.781316 (+ 538us) server_negotiation.cc:206] Beginning negotiation
0810 03:26:14.781320 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:26:14.782595 (+ 1275us) server_negotiation.cc:407] Connection header received
0810 03:26:14.785080 (+ 2485us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:26:14.785087 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:26:14.785176 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:26:14.785267 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:26:14.786923 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:14.787739 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:14.789389 (+ 1650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:14.789811 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:14.789921 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:14.794198 (+ 4277us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:26:14.794207 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:26:14.794211 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:26:14.794267 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:26:14.798036 (+ 3769us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:26:14.799007 (+ 971us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:26:14.799012 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:26:14.799015 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:26:14.799159 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:26:14.799604 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:26:14.799609 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:26:14.799611 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:26:14.799938 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:26:14.800058 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0810 03:26:14.800870 (+ 812us) server_negotiation.cc:299] Negotiation successful
0810 03:26:14.801093 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":421,"thread_start_us":268,"threads_started":1}
I20260810 03:26:14.802342 1949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:14.781474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:36881 (local address 127.27.161.3:39561)
0810 03:26:14.782426 (+ 952us) negotiation.cc:107] Waiting for socket to connect
0810 03:26:14.782473 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0810 03:26:14.784746 (+ 2273us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:26:14.785482 (+ 736us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:26:14.785491 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:26:14.786120 (+ 629us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:26:14.786734 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:26:14.786753 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:26:14.787906 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:26:14.787911 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:26:14.789217 (+ 1306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:26:14.789233 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:26:14.790216 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:26:14.790283 (+ 67us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:26:14.790579 (+ 296us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:14.791398 (+ 819us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:26:14.791438 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:26:14.793803 (+ 2365us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:26:14.798245 (+ 4442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:26:14.798254 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:26:14.798271 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:26:14.798834 (+ 563us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:26:14.799246 (+ 412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:26:14.799249 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:26:14.799252 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:26:14.799453 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:26:14.800048 (+ 595us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:26:14.800054 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:26:14.800420 (+ 366us) client_negotiation.cc:769] Sending connection context
0810 03:26:14.800692 (+ 272us) client_negotiation.cc:240] Negotiation successful
0810 03:26:14.801126 (+ 434us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":644,"thread_start_us":368,"threads_started":1}
I20260810 03:26:14.803823 1946 heartbeater.cc:347] Connected to a master server at 127.27.161.62:36881
I20260810 03:26:14.804153 1946 heartbeater.cc:464] Registering TS with master...
I20260810 03:26:14.804936 1946 heartbeater.cc:511] Master 127.27.161.62:36881 requested a full tablet report, sending...
I20260810 03:26:14.806558 1279 ts_manager.cc:194] Registered new tserver with Master: 4c2e595f4c71400793c6588195eac760 (127.27.161.3:38839)
I20260810 03:26:14.807404 1279 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.161.3@KRBTEST.COM'} at 127.27.161.3:39561
I20260810 03:26:14.810619 28292 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 10 03:26:14 dist-test-slave-vpvm krb5kdc[1209](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332374, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 10 03:26:14 dist-test-slave-vpvm krb5kdc[1209](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332374, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.161.1@KRBTEST.COM
I20260810 03:26:14.871959 1955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:14.854837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57550 (local address 127.27.161.1:41029)
0810 03:26:14.856017 (+ 1180us) server_negotiation.cc:206] Beginning negotiation
0810 03:26:14.856024 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:26:14.856051 (+ 27us) server_negotiation.cc:407] Connection header received
0810 03:26:14.856114 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:26:14.856121 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:26:14.856338 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:26:14.856472 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:26:14.858093 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:14.859167 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:14.860528 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:14.861135 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:14.861240 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:14.865439 (+ 4199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:26:14.865458 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:26:14.865476 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:26:14.865525 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:26:14.868572 (+ 3047us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:26:14.869396 (+ 824us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:26:14.869406 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:26:14.869408 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:26:14.869599 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:26:14.870159 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:26:14.870164 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:26:14.870166 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:26:14.870588 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:26:14.870846 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0810 03:26:14.871292 (+ 446us) server_negotiation.cc:299] Negotiation successful
0810 03:26:14.871600 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1007,"thread_start_us":856,"threads_started":1}
Aug 10 03:26:14 dist-test-slave-vpvm krb5kdc[1209](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332374, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.161.62@KRBTEST.COM
I20260810 03:26:14.892557 1948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:14.877369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33352 (local address 127.27.161.62:36881)
0810 03:26:14.877595 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0810 03:26:14.877601 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:26:14.877854 (+ 253us) server_negotiation.cc:407] Connection header received
0810 03:26:14.878016 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:26:14.878022 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:26:14.878103 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:26:14.878190 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:26:14.879599 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:14.880379 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:14.881759 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:14.882249 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:14.882358 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:14.886738 (+ 4380us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:26:14.886744 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:26:14.886748 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:26:14.886797 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:26:14.889531 (+ 2734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:26:14.890445 (+ 914us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:26:14.890449 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:26:14.890452 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:26:14.890602 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:26:14.891043 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:26:14.891047 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:26:14.891050 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:26:14.891441 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:26:14.891659 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0810 03:26:14.892017 (+ 358us) server_negotiation.cc:299] Negotiation successful
0810 03:26:14.892233 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
W20260810 03:26:14.893151 1279 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33352
I20260810 03:26:14.894246 28292 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iBH5/build/asan/bin/kudu pid 1361
I20260810 03:26:14.907130 1955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:14.895156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57562 (local address 127.27.161.1:41029)
0810 03:26:14.895396 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0810 03:26:14.895402 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:26:14.895682 (+ 280us) server_negotiation.cc:407] Connection header received
0810 03:26:14.895814 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:26:14.895817 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:26:14.895893 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:26:14.895980 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:26:14.897270 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:14.898134 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:14.899384 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:14.899853 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:14.899989 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:14.902297 (+ 2308us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:26:14.902302 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:26:14.902306 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:26:14.902345 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:26:14.904867 (+ 2522us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:26:14.905457 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:26:14.905460 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:26:14.905462 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:26:14.905661 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:26:14.905975 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:26:14.905979 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:26:14.905982 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:26:14.906287 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:26:14.906470 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0810 03:26:14.906715 (+ 245us) server_negotiation.cc:299] Negotiation successful
0810 03:26:14.906883 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260810 03:26:14.907749 1537 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:26:14.991194 28292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iBH5/build/asan/bin/kudu with pid 1361
I20260810 03:26:15.015797 28292 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iBH5/build/asan/bin/kudu pid 1553
Aug 10 03:26:15 dist-test-slave-vpvm krb5kdc[1209](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332374, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.161.2@KRBTEST.COM
I20260810 03:26:15.033177 1961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:15.016854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37456 (local address 127.27.161.2:45459)
0810 03:26:15.017420 (+ 566us) server_negotiation.cc:206] Beginning negotiation
0810 03:26:15.017427 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:26:15.017454 (+ 27us) server_negotiation.cc:407] Connection header received
0810 03:26:15.017517 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:26:15.017524 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:26:15.017792 (+ 268us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:26:15.018026 (+ 234us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:26:15.019384 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:15.020395 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:15.021681 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:15.022290 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:15.022414 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:15.026910 (+ 4496us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:26:15.026929 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:26:15.026948 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:26:15.026999 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:26:15.030312 (+ 3313us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:26:15.031080 (+ 768us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:26:15.031090 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:26:15.031092 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:26:15.031239 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:26:15.031578 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:26:15.031582 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:26:15.031585 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:26:15.031998 (+ 413us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:26:15.032263 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0810 03:26:15.032517 (+ 254us) server_negotiation.cc:299] Negotiation successful
0810 03:26:15.032749 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":406,"thread_start_us":256,"threads_started":1}
I20260810 03:26:15.034704 1740 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:26:15.116675 28292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iBH5/build/asan/bin/kudu with pid 1553
I20260810 03:26:15.140834 28292 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iBH5/build/asan/bin/kudu pid 1760
Aug 10 03:26:15 dist-test-slave-vpvm krb5kdc[1209](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332374, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.161.3@KRBTEST.COM
I20260810 03:26:15.160679 1963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:15.141983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35632 (local address 127.27.161.3:38839)
0810 03:26:15.142649 (+ 666us) server_negotiation.cc:206] Beginning negotiation
0810 03:26:15.142657 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 03:26:15.142687 (+ 30us) server_negotiation.cc:407] Connection header received
0810 03:26:15.142753 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:26:15.142760 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:26:15.142969 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:26:15.143106 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:26:15.144369 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:15.145449 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:15.146760 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:15.147326 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:15.147426 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:15.154423 (+ 6997us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:26:15.154441 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:26:15.154461 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:26:15.154512 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:26:15.157488 (+ 2976us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:26:15.158363 (+ 875us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:26:15.158372 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:26:15.158374 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:26:15.158611 (+ 237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:26:15.159027 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:26:15.159032 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:26:15.159035 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:26:15.159427 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:26:15.159693 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0810 03:26:15.160069 (+ 376us) server_negotiation.cc:299] Negotiation successful
0810 03:26:15.160338 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":502,"thread_start_us":338,"threads_started":1}
I20260810 03:26:15.162179 1938 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:26:15.246093 28292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iBH5/build/asan/bin/kudu with pid 1760
I20260810 03:26:15.266108 28292 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iBH5/build/asan/bin/kudu pid 1229
I20260810 03:26:15.267375 1323 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:26:15.375471 28292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iBH5/build/asan/bin/kudu with pid 1229
2026-08-10T03:26:15Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (4291 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786332336319544-28292-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 03:26:15 dist-test-slave-vpvm krb5kdc[1971](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 03:26:15 dist-test-slave-vpvm krb5kdc[1971](info): set up 2 sockets
Aug 10 03:26:15 dist-test-slave-vpvm krb5kdc[1971](info): commencing operation
krb5kdc: starting...
W20260810 03:26:16.606200 28292 mini_kdc.cc:121] Time spent starting KDC: real 1.178s user 0.000s sys 0.037s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786332336319544-28292-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786332336319544-28292-0/krb5kdc/test-user.keytab.
Aug 10 03:26:16 dist-test-slave-vpvm krb5kdc[1971](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332376, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T03:26:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T03:26:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.161.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.161.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.161.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.62.keytab.
Entry for principal kudu/127.27.161.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.62.keytab.
I20260810 03:26:16.858428 28292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iBH5/build/asan/bin/kudu
/tmp/dist-test-task24iBH5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786332336319544-28292-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786332336319544-28292-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786332336319544-28292-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786332336319544-28292-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.161.62:33451
--webserver_interface=127.27.161.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.62.keytab
--principal=kudu/127.27.161.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.161.20:41981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.161.62:33451
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786332336319544-28292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786332336319544-28292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786332336319544-28292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:26:17.209641 1989 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:26:17.210042 1989 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:26:17.210166 1989 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:26:17.217862 1989 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:26:17.218050 1989 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:26:17.218120 1989 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:26:17.218181 1989 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:26:17.218230 1989 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:26:17.228907 1989 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.161.20:41981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786332336319544-28292-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786332336319544-28292-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.161.62:33451
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.62.keytab
--principal=kudu/127.27.161.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.161.62:33451
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786332336319544-28292-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.161.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786332336319544-28292-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:26:17.230333 1989 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:26:17.231715 1989 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:26:17.244166 2004 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:26:17.244212 2001 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:26:17.244243 2000 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:26:17.245158 1989 server_base.cc:1034] running on GCE node
I20260810 03:26:17.246112 1989 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:26:17.247713 1989 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:26:17.248955 1989 hybrid_clock.cc:648] HybridClock initialized: now 1786332377248896 us; error 59 us; skew 500 ppm
I20260810 03:26:17.249354 1989 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:26:17 dist-test-slave-vpvm krb5kdc[1971](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332377, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.161.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:26:17.252817 1989 init.cc:377] Logged in from keytab as kudu/127.27.161.62@KRBTEST.COM (short username kudu)
I20260810 03:26:17.254241 1989 fs_manager.cc:359] Metadata directory not provided
I20260810 03:26:17.254364 1989 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:26:17.254703 1989 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:26:17.258065 1989 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786332336319544-28292-0/minicluster-data/master-0/data/instance:
uuid: "79ba9581131e4f2ba9d01474d4d54360"
format_stamp: "Formatted at 2026-08-10 03:26:17 on dist-test-slave-vpvm"
server_key: "1bf1ef007dc45733252d116b9e50d103"
server_key_iv: "661f0a7fd53712c621c32010f72c26ad"
server_key_version: "encryptionkey@0"
I20260810 03:26:17.259004 1989 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786332336319544-28292-0/minicluster-data/master-0/wal/instance:
uuid: "79ba9581131e4f2ba9d01474d4d54360"
format_stamp: "Formatted at 2026-08-10 03:26:17 on dist-test-slave-vpvm"
server_key: "1bf1ef007dc45733252d116b9e50d103"
server_key_iv: "661f0a7fd53712c621c32010f72c26ad"
server_key_version: "encryptionkey@0"
I20260810 03:26:17.264727 1989 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260810 03:26:17.268568 2015 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:26:17.270025 1989 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 03:26:17.270221 1989 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786332336319544-28292-0/minicluster-data/master-0/data,/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786332336319544-28292-0/minicluster-data/master-0/wal
uuid: "79ba9581131e4f2ba9d01474d4d54360"
format_stamp: "Formatted at 2026-08-10 03:26:17 on dist-test-slave-vpvm"
server_key: "1bf1ef007dc45733252d116b9e50d103"
server_key_iv: "661f0a7fd53712c621c32010f72c26ad"
server_key_version: "encryptionkey@0"
I20260810 03:26:17.270447 1989 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786332336319544-28292-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786332336319544-28292-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786332336319544-28292-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:26:17.290787 1989 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:26:17.296015 1989 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:26:17.296293 1989 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:26:17.315457 1989 rpc_server.cc:307] RPC server started. Bound to: 127.27.161.62:33451
I20260810 03:26:17.315469 2100 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.161.62:33451 every 8 connection(s)
I20260810 03:26:17.316464 1989 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786332336319544-28292-0/minicluster-data/master-0/data/info.pb
I20260810 03:26:17.320789 2101 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:26:17.324762 28292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iBH5/build/asan/bin/kudu as pid 1989
I20260810 03:26:17.325023 28292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786332336319544-28292-0/minicluster-data/master-0/wal/instance
I20260810 03:26:17.325824 28292 external_mini_cluster.cc:1428] Setting key 31dbc52a57ee7d190f073b41b47afb29
I20260810 03:26:17.336341 2101 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 79ba9581131e4f2ba9d01474d4d54360: Bootstrap starting.
I20260810 03:26:17.341037 2101 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 79ba9581131e4f2ba9d01474d4d54360: Neither blocks nor log segments found. Creating new log.
I20260810 03:26:17.342598 2101 log.cc:824] T 00000000000000000000000000000000 P 79ba9581131e4f2ba9d01474d4d54360: Log is configured to *not* fsync() on all Append() calls
Aug 10 03:26:17 dist-test-slave-vpvm krb5kdc[1971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332376, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.161.62@KRBTEST.COM
I20260810 03:26:17.346212 2101 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 79ba9581131e4f2ba9d01474d4d54360: No bootstrap required, opened a new log
I20260810 03:26:17.352262 2104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:17.328236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33864 (local address 127.27.161.62:33451)
0810 03:26:17.329214 (+ 978us) server_negotiation.cc:206] Beginning negotiation
0810 03:26:17.329226 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 03:26:17.329273 (+ 47us) server_negotiation.cc:407] Connection header received
0810 03:26:17.331027 (+ 1754us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:26:17.331058 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:26:17.331595 (+ 537us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:26:17.331982 (+ 387us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:26:17.334655 (+ 2673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:17.336212 (+ 1557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:17.338170 (+ 1958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:17.338972 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:17.339144 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:17.344516 (+ 5372us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:26:17.344526 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:26:17.344544 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:26:17.344592 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:26:17.347796 (+ 3204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:26:17.348582 (+ 786us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:26:17.348589 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:26:17.348596 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:26:17.348779 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:26:17.349216 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:26:17.349221 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:26:17.349223 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:26:17.349850 (+ 627us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:26:17.350200 (+ 350us) server_negotiation.cc:1035] Waiting for connection context
0810 03:26:17.350712 (+ 512us) server_negotiation.cc:299] Negotiation successful
0810 03:26:17.351161 (+ 449us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":753,"thread_start_us":470,"threads_started":1}
I20260810 03:26:17.354209 2101 raft_consensus.cc:348] T 00000000000000000000000000000000 P 79ba9581131e4f2ba9d01474d4d54360 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79ba9581131e4f2ba9d01474d4d54360" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 33451 } }
I20260810 03:26:17.354580 2101 raft_consensus.cc:374] T 00000000000000000000000000000000 P 79ba9581131e4f2ba9d01474d4d54360 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:26:17.354674 2101 raft_consensus.cc:729] T 00000000000000000000000000000000 P 79ba9581131e4f2ba9d01474d4d54360 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79ba9581131e4f2ba9d01474d4d54360, State: Initialized, Role: FOLLOWER
I20260810 03:26:17.355261 2101 consensus_queue.cc:260] T 00000000000000000000000000000000 P 79ba9581131e4f2ba9d01474d4d54360 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79ba9581131e4f2ba9d01474d4d54360" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 33451 } }
I20260810 03:26:17.355504 2101 raft_consensus.cc:388] T 00000000000000000000000000000000 P 79ba9581131e4f2ba9d01474d4d54360 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 03:26:17.355638 2101 raft_consensus.cc:482] T 00000000000000000000000000000000 P 79ba9581131e4f2ba9d01474d4d54360 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 03:26:17.355810 2101 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 79ba9581131e4f2ba9d01474d4d54360 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:26:17.358155 2101 raft_consensus.cc:504] T 00000000000000000000000000000000 P 79ba9581131e4f2ba9d01474d4d54360 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79ba9581131e4f2ba9d01474d4d54360" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 33451 } }
I20260810 03:26:17.358716 2101 leader_election.cc:302] T 00000000000000000000000000000000 P 79ba9581131e4f2ba9d01474d4d54360 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 79ba9581131e4f2ba9d01474d4d54360; no voters:
I20260810 03:26:17.359319 2101 leader_election.cc:288] T 00000000000000000000000000000000 P 79ba9581131e4f2ba9d01474d4d54360 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 03:26:17.359510 2106 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 79ba9581131e4f2ba9d01474d4d54360 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:26:17.360296 2106 raft_consensus.cc:686] T 00000000000000000000000000000000 P 79ba9581131e4f2ba9d01474d4d54360 [term 1 LEADER]: Becoming Leader. State: Replica: 79ba9581131e4f2ba9d01474d4d54360, State: Running, Role: LEADER
I20260810 03:26:17.361483 2101 sys_catalog.cc:564] T 00000000000000000000000000000000 P 79ba9581131e4f2ba9d01474d4d54360 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:26:17.361773 2106 consensus_queue.cc:237] T 00000000000000000000000000000000 P 79ba9581131e4f2ba9d01474d4d54360 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79ba9581131e4f2ba9d01474d4d54360" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 33451 } }
I20260810 03:26:17.364845 2110 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79ba9581131e4f2ba9d01474d4d54360 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "79ba9581131e4f2ba9d01474d4d54360" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79ba9581131e4f2ba9d01474d4d54360" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 33451 } } }
I20260810 03:26:17.365294 2110 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79ba9581131e4f2ba9d01474d4d54360 [sys.catalog]: This master's current role is: LEADER
I20260810 03:26:17.365825 2111 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79ba9581131e4f2ba9d01474d4d54360 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 79ba9581131e4f2ba9d01474d4d54360. Latest consensus state: current_term: 1 leader_uuid: "79ba9581131e4f2ba9d01474d4d54360" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79ba9581131e4f2ba9d01474d4d54360" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 33451 } } }
I20260810 03:26:17.366200 2111 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79ba9581131e4f2ba9d01474d4d54360 [sys.catalog]: This master's current role is: LEADER
I20260810 03:26:17.367985 2122 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 03:26:17.372840 2122 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 03:26:17.383033 2122 catalog_manager.cc:1347] Generated new cluster ID: 3bbf3a9ed5454c66ac40be645d39aa9d
I20260810 03:26:17.383183 2122 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 03:26:17.391734 2122 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 03:26:17.393174 2122 catalog_manager.cc:1504] Loading token signing keys...
I20260810 03:26:17.409096 2122 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 79ba9581131e4f2ba9d01474d4d54360: Generated new TSK 0
I20260810 03:26:17.410121 2122 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.161.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.161.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.161.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.1.keytab.
Entry for principal kudu/127.27.161.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.1.keytab.
I20260810 03:26:17.479444 28292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iBH5/build/asan/bin/kudu
/tmp/dist-test-task24iBH5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786332336319544-28292-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786332336319544-28292-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786332336319544-28292-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786332336319544-28292-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.161.1:0
--local_ip_for_outbound_sockets=127.27.161.1
--webserver_interface=127.27.161.1
--webserver_port=0
--tserver_master_addrs=127.27.161.62:33451
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.1.keytab
--principal=kudu/127.27.161.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.161.20:41981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786332336319544-28292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786332336319544-28292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786332336319544-28292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:26:17.831831 2135 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:26:17.832144 2135 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:26:17.832304 2135 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:26:17.839921 2135 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:26:17.840041 2135 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:26:17.840162 2135 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.161.1
I20260810 03:26:17.850864 2135 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.161.20:41981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786332336319544-28292-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786332336319544-28292-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.1.keytab
--principal=kudu/127.27.161.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.161.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786332336319544-28292-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.161.1
--webserver_port=0
--tserver_master_addrs=127.27.161.62:33451
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.161.1
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786332336319544-28292-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:26:17.852288 2135 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:26:17.853710 2135 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:26:17.867499 2145 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:26:17.867733 2144 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:26:17.867543 2147 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:26:17.868455 2135 server_base.cc:1034] running on GCE node
I20260810 03:26:17.869143 2135 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:26:17.870250 2135 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:26:17.871502 2135 hybrid_clock.cc:648] HybridClock initialized: now 1786332377871441 us; error 54 us; skew 500 ppm
I20260810 03:26:17.871865 2135 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:26:17 dist-test-slave-vpvm krb5kdc[1971](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332377, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.161.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:26:17.875334 2135 init.cc:377] Logged in from keytab as kudu/127.27.161.1@KRBTEST.COM (short username kudu)
I20260810 03:26:17.876569 2135 fs_manager.cc:359] Metadata directory not provided
I20260810 03:26:17.876690 2135 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:26:17.876976 2135 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:26:17.880160 2135 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786332336319544-28292-0/minicluster-data/ts-0/data/instance:
uuid: "e5c76f8b11ba464d9c9d2c878fe4119a"
format_stamp: "Formatted at 2026-08-10 03:26:17 on dist-test-slave-vpvm"
server_key: "ea2cedb504dab35f06f8a40865434b4b"
server_key_iv: "d7826b710e6ab27d0fa8bcd3f76c3925"
server_key_version: "encryptionkey@0"
I20260810 03:26:17.881003 2135 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786332336319544-28292-0/minicluster-data/ts-0/wal/instance:
uuid: "e5c76f8b11ba464d9c9d2c878fe4119a"
format_stamp: "Formatted at 2026-08-10 03:26:17 on dist-test-slave-vpvm"
server_key: "ea2cedb504dab35f06f8a40865434b4b"
server_key_iv: "d7826b710e6ab27d0fa8bcd3f76c3925"
server_key_version: "encryptionkey@0"
I20260810 03:26:17.886451 2135 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260810 03:26:17.890202 2154 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:26:17.891662 2135 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 03:26:17.891844 2135 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786332336319544-28292-0/minicluster-data/ts-0/data,/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786332336319544-28292-0/minicluster-data/ts-0/wal
uuid: "e5c76f8b11ba464d9c9d2c878fe4119a"
format_stamp: "Formatted at 2026-08-10 03:26:17 on dist-test-slave-vpvm"
server_key: "ea2cedb504dab35f06f8a40865434b4b"
server_key_iv: "d7826b710e6ab27d0fa8bcd3f76c3925"
server_key_version: "encryptionkey@0"
I20260810 03:26:17.892084 2135 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786332336319544-28292-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786332336319544-28292-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786332336319544-28292-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:26:17.909611 2135 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:26:17.914952 2135 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:26:17.915252 2135 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:26:17.916467 2135 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:26:17.918181 2135 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:26:17.918288 2135 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:26:17.918397 2135 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:26:17.918473 2135 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:26:17.951709 2135 rpc_server.cc:307] RPC server started. Bound to: 127.27.161.1:44809
I20260810 03:26:17.951778 2322 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.161.1:44809 every 8 connection(s)
I20260810 03:26:17.952768 2135 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786332336319544-28292-0/minicluster-data/ts-0/data/info.pb
I20260810 03:26:17.955029 28292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iBH5/build/asan/bin/kudu as pid 2135
I20260810 03:26:17.955276 28292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786332336319544-28292-0/minicluster-data/ts-0/wal/instance
I20260810 03:26:17.956128 28292 external_mini_cluster.cc:1428] Setting key c006c79f2ef099752cd28e224f696161
Aug 10 03:26:17 dist-test-slave-vpvm krb5kdc[1971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332377, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.161.1@KRBTEST.COM for kudu/127.27.161.62@KRBTEST.COM
I20260810 03:26:17.976127 2327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:17.955591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.161.1:51479 (local address 127.27.161.62:33451)
0810 03:26:17.956034 (+ 443us) server_negotiation.cc:206] Beginning negotiation
0810 03:26:17.956039 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:26:17.957410 (+ 1371us) server_negotiation.cc:407] Connection header received
0810 03:26:17.959618 (+ 2208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:26:17.959624 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:26:17.959706 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:26:17.959814 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:26:17.961428 (+ 1614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:17.962434 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:17.964253 (+ 1819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:17.964817 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:17.964950 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:17.969033 (+ 4083us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:26:17.969040 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:26:17.969045 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:26:17.969096 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:26:17.972299 (+ 3203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:26:17.973238 (+ 939us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:26:17.973245 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:26:17.973247 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:26:17.973406 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:26:17.973980 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:26:17.973986 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:26:17.973990 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:26:17.974352 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:26:17.974515 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0810 03:26:17.975499 (+ 984us) server_negotiation.cc:299] Negotiation successful
0810 03:26:17.975778 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":336,"thread_start_us":228,"threads_started":1}
I20260810 03:26:17.977267 2329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:17.956328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:33451 (local address 127.27.161.1:51479)
0810 03:26:17.957265 (+ 937us) negotiation.cc:107] Waiting for socket to connect
0810 03:26:17.957306 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0810 03:26:17.959327 (+ 2021us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:26:17.960016 (+ 689us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:26:17.960027 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:26:17.960588 (+ 561us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:26:17.961233 (+ 645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:26:17.961260 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:26:17.962660 (+ 1400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:26:17.962665 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:26:17.964078 (+ 1413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:26:17.964093 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:26:17.965087 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:26:17.965093 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:26:17.965315 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:17.966112 (+ 797us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:26:17.966140 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:26:17.968744 (+ 2604us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:26:17.972526 (+ 3782us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:26:17.972535 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:26:17.972552 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:26:17.973092 (+ 540us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:26:17.973568 (+ 476us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:26:17.973573 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:26:17.973575 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:26:17.973812 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:26:17.974519 (+ 707us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:26:17.974527 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:26:17.974929 (+ 402us) client_negotiation.cc:769] Sending connection context
0810 03:26:17.975321 (+ 392us) client_negotiation.cc:240] Negotiation successful
0810 03:26:17.975988 (+ 667us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":651,"thread_start_us":408,"threads_started":1}
I20260810 03:26:17.979470 2323 heartbeater.cc:347] Connected to a master server at 127.27.161.62:33451
I20260810 03:26:17.979830 2323 heartbeater.cc:464] Registering TS with master...
I20260810 03:26:17.980594 2323 heartbeater.cc:511] Master 127.27.161.62:33451 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.27.161.2@KRBTEST.COM; defaulting to no policy
I20260810 03:26:17.983031 2047 ts_manager.cc:194] Registered new tserver with Master: e5c76f8b11ba464d9c9d2c878fe4119a (127.27.161.1:44809)
I20260810 03:26:17.984630 2047 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.161.1@KRBTEST.COM'} at 127.27.161.1:51479
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.161.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.161.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.2.keytab.
Entry for principal kudu/127.27.161.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.2.keytab.
I20260810 03:26:18.015509 28292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iBH5/build/asan/bin/kudu
/tmp/dist-test-task24iBH5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786332336319544-28292-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786332336319544-28292-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786332336319544-28292-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786332336319544-28292-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.161.2:0
--local_ip_for_outbound_sockets=127.27.161.2
--webserver_interface=127.27.161.2
--webserver_port=0
--tserver_master_addrs=127.27.161.62:33451
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.2.keytab
--principal=kudu/127.27.161.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.161.20:41981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786332336319544-28292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786332336319544-28292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786332336319544-28292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:26:18.370319 2333 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:26:18.370695 2333 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:26:18.370834 2333 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:26:18.378803 2333 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:26:18.379108 2333 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:26:18.379300 2333 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.161.2
I20260810 03:26:18.389822 2333 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.161.20:41981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786332336319544-28292-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786332336319544-28292-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.2.keytab
--principal=kudu/127.27.161.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.161.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786332336319544-28292-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.161.2
--webserver_port=0
--tserver_master_addrs=127.27.161.62:33451
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.161.2
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786332336319544-28292-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:26:18.391347 2333 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:26:18.392725 2333 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:26:18.406517 2343 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:26:18.406611 2340 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:26:18.406558 2341 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:26:18.407114 2333 server_base.cc:1034] running on GCE node
I20260810 03:26:18.407806 2333 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:26:18.408825 2333 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:26:18.410101 2333 hybrid_clock.cc:648] HybridClock initialized: now 1786332378410029 us; error 60 us; skew 500 ppm
I20260810 03:26:18.410459 2333 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:26:18 dist-test-slave-vpvm krb5kdc[1971](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332378, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.161.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:26:18.414052 2333 init.cc:377] Logged in from keytab as kudu/127.27.161.2@KRBTEST.COM (short username kudu)
I20260810 03:26:18.415475 2333 fs_manager.cc:359] Metadata directory not provided
I20260810 03:26:18.415629 2333 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:26:18.416045 2333 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:26:18.419970 2333 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786332336319544-28292-0/minicluster-data/ts-1/data/instance:
uuid: "a93b5c66a3de4c0f995bec49133af2f5"
format_stamp: "Formatted at 2026-08-10 03:26:18 on dist-test-slave-vpvm"
server_key: "aafd5a68bbc82078210d138eca77d947"
server_key_iv: "1e3cbc9c6aca4b83be396635dcd1192e"
server_key_version: "encryptionkey@0"
I20260810 03:26:18.420854 2333 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786332336319544-28292-0/minicluster-data/ts-1/wal/instance:
uuid: "a93b5c66a3de4c0f995bec49133af2f5"
format_stamp: "Formatted at 2026-08-10 03:26:18 on dist-test-slave-vpvm"
server_key: "aafd5a68bbc82078210d138eca77d947"
server_key_iv: "1e3cbc9c6aca4b83be396635dcd1192e"
server_key_version: "encryptionkey@0"
I20260810 03:26:18.426609 2333 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260810 03:26:18.430725 2355 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:26:18.432535 2333 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260810 03:26:18.432755 2333 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786332336319544-28292-0/minicluster-data/ts-1/data,/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786332336319544-28292-0/minicluster-data/ts-1/wal
uuid: "a93b5c66a3de4c0f995bec49133af2f5"
format_stamp: "Formatted at 2026-08-10 03:26:18 on dist-test-slave-vpvm"
server_key: "aafd5a68bbc82078210d138eca77d947"
server_key_iv: "1e3cbc9c6aca4b83be396635dcd1192e"
server_key_version: "encryptionkey@0"
I20260810 03:26:18.433010 2333 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786332336319544-28292-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786332336319544-28292-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786332336319544-28292-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:26:18.458287 2333 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:26:18.463155 2333 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:26:18.463459 2333 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:26:18.464624 2333 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:26:18.466257 2333 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:26:18.466367 2333 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:26:18.466486 2333 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:26:18.466559 2333 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:26:18.499063 2333 rpc_server.cc:307] RPC server started. Bound to: 127.27.161.2:45177
I20260810 03:26:18.499117 2528 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.161.2:45177 every 8 connection(s)
I20260810 03:26:18.499954 2333 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786332336319544-28292-0/minicluster-data/ts-1/data/info.pb
I20260810 03:26:18.503625 28292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iBH5/build/asan/bin/kudu as pid 2333
I20260810 03:26:18.503835 28292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786332336319544-28292-0/minicluster-data/ts-1/wal/instance
I20260810 03:26:18.504315 28292 external_mini_cluster.cc:1428] Setting key 80d7704291e20a520b2739a4e05df36d
Aug 10 03:26:18 dist-test-slave-vpvm krb5kdc[1971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332378, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.161.2@KRBTEST.COM for kudu/127.27.161.62@KRBTEST.COM
I20260810 03:26:18.521308 2531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:18.502591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.161.2:55355 (local address 127.27.161.62:33451)
0810 03:26:18.503071 (+ 480us) server_negotiation.cc:206] Beginning negotiation
0810 03:26:18.503076 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:26:18.504224 (+ 1148us) server_negotiation.cc:407] Connection header received
0810 03:26:18.506318 (+ 2094us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:26:18.506323 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:26:18.506406 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:26:18.506505 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:26:18.508051 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:18.508792 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:18.510460 (+ 1668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:18.510910 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:18.511026 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:18.514999 (+ 3973us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:26:18.515005 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:26:18.515009 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:26:18.515049 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:26:18.517993 (+ 2944us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:26:18.518843 (+ 850us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:26:18.518848 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:26:18.518850 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:26:18.518987 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:26:18.519433 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:26:18.519437 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:26:18.519439 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:26:18.519737 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:26:18.519877 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0810 03:26:18.520629 (+ 752us) server_negotiation.cc:299] Negotiation successful
0810 03:26:18.520898 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":365,"thread_start_us":243,"threads_started":1}
I20260810 03:26:18.522404 2533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:18.503188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:33451 (local address 127.27.161.2:55355)
0810 03:26:18.504020 (+ 832us) negotiation.cc:107] Waiting for socket to connect
0810 03:26:18.504070 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0810 03:26:18.506047 (+ 1977us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:26:18.506752 (+ 705us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:26:18.506761 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:26:18.507299 (+ 538us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:26:18.507873 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:26:18.507891 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:26:18.508953 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:26:18.508957 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:26:18.510312 (+ 1355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:26:18.510326 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:26:18.511157 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:26:18.511165 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:26:18.511386 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:18.512110 (+ 724us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:26:18.512135 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:26:18.514713 (+ 2578us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:26:18.518178 (+ 3465us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:26:18.518189 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:26:18.518207 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:26:18.518707 (+ 500us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:26:18.519102 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:26:18.519107 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:26:18.519111 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:26:18.519308 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:26:18.519870 (+ 562us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:26:18.519879 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:26:18.520250 (+ 371us) client_negotiation.cc:769] Sending connection context
0810 03:26:18.520504 (+ 254us) client_negotiation.cc:240] Negotiation successful
0810 03:26:18.521119 (+ 615us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":548,"thread_start_us":332,"threads_started":1}
I20260810 03:26:18.524130 2529 heartbeater.cc:347] Connected to a master server at 127.27.161.62:33451
I20260810 03:26:18.524523 2529 heartbeater.cc:464] Registering TS with master...
I20260810 03:26:18.525416 2529 heartbeater.cc:511] Master 127.27.161.62:33451 requested a full tablet report, sending...
I20260810 03:26:18.527456 2047 ts_manager.cc:194] Registered new tserver with Master: a93b5c66a3de4c0f995bec49133af2f5 (127.27.161.2:45177)
I20260810 03:26:18.528746 2047 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.161.2@KRBTEST.COM'} at 127.27.161.2:55355
WARNING: no policy specified for kudu/127.27.161.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.161.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.161.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.3.keytab.
Entry for principal kudu/127.27.161.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.3.keytab.
I20260810 03:26:18.561448 28292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iBH5/build/asan/bin/kudu
/tmp/dist-test-task24iBH5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786332336319544-28292-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786332336319544-28292-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786332336319544-28292-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786332336319544-28292-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.161.3:0
--local_ip_for_outbound_sockets=127.27.161.3
--webserver_interface=127.27.161.3
--webserver_port=0
--tserver_master_addrs=127.27.161.62:33451
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.3.keytab
--principal=kudu/127.27.161.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.161.20:41981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786332336319544-28292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786332336319544-28292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786332336319544-28292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:26:18.917876 2536 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:26:18.918234 2536 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:26:18.918339 2536 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:26:18.925685 2536 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:26:18.925809 2536 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:26:18.925932 2536 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.161.3
I20260810 03:26:18.936246 2536 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.161.20:41981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786332336319544-28292-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786332336319544-28292-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.3.keytab
--principal=kudu/127.27.161.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.161.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786332336319544-28292-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.161.3
--webserver_port=0
--tserver_master_addrs=127.27.161.62:33451
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.161.3
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786332336319544-28292-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:26:18.937709 2536 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:26:18.939163 2536 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:26:18.952751 2546 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:26:18.952720 2547 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:26:18.953040 2550 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:26:18.954905 2536 server_base.cc:1034] running on GCE node
I20260810 03:26:18.955509 2536 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:26:18.956475 2536 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:26:18.957688 2536 hybrid_clock.cc:648] HybridClock initialized: now 1786332378957647 us; error 45 us; skew 500 ppm
I20260810 03:26:18.958039 2536 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:26:18 dist-test-slave-vpvm krb5kdc[1971](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332378, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.161.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:26:18.961067 2536 init.cc:377] Logged in from keytab as kudu/127.27.161.3@KRBTEST.COM (short username kudu)
I20260810 03:26:18.962316 2536 fs_manager.cc:359] Metadata directory not provided
I20260810 03:26:18.962423 2536 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:26:18.962778 2536 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:26:18.966053 2536 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786332336319544-28292-0/minicluster-data/ts-2/data/instance:
uuid: "17729f053c294195a3f6c51ee83e12cc"
format_stamp: "Formatted at 2026-08-10 03:26:18 on dist-test-slave-vpvm"
server_key: "1efe187297950e69be6447d4198ee160"
server_key_iv: "be65037d65437e2dce3bb953a4ed65a5"
server_key_version: "encryptionkey@0"
I20260810 03:26:18.966921 2536 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786332336319544-28292-0/minicluster-data/ts-2/wal/instance:
uuid: "17729f053c294195a3f6c51ee83e12cc"
format_stamp: "Formatted at 2026-08-10 03:26:18 on dist-test-slave-vpvm"
server_key: "1efe187297950e69be6447d4198ee160"
server_key_iv: "be65037d65437e2dce3bb953a4ed65a5"
server_key_version: "encryptionkey@0"
I20260810 03:26:18.972328 2536 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260810 03:26:18.976208 2559 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:26:18.977665 2536 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 03:26:18.977851 2536 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786332336319544-28292-0/minicluster-data/ts-2/data,/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786332336319544-28292-0/minicluster-data/ts-2/wal
uuid: "17729f053c294195a3f6c51ee83e12cc"
format_stamp: "Formatted at 2026-08-10 03:26:18 on dist-test-slave-vpvm"
server_key: "1efe187297950e69be6447d4198ee160"
server_key_iv: "be65037d65437e2dce3bb953a4ed65a5"
server_key_version: "encryptionkey@0"
I20260810 03:26:18.978138 2536 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786332336319544-28292-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786332336319544-28292-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786332336319544-28292-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:26:18.988631 2323 heartbeater.cc:503] Master 127.27.161.62:33451 was elected leader, sending a full tablet report...
I20260810 03:26:19.010226 2536 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:26:19.014721 2536 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:26:19.015017 2536 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:26:19.016075 2536 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:26:19.017558 2536 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:26:19.017658 2536 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:26:19.017773 2536 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:26:19.017843 2536 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:26:19.052588 2536 rpc_server.cc:307] RPC server started. Bound to: 127.27.161.3:46821
I20260810 03:26:19.052649 2730 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.161.3:46821 every 8 connection(s)
I20260810 03:26:19.053486 2536 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786332336319544-28292-0/minicluster-data/ts-2/data/info.pb
I20260810 03:26:19.059212 28292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iBH5/build/asan/bin/kudu as pid 2536
I20260810 03:26:19.059414 28292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786332336319544-28292-0/minicluster-data/ts-2/wal/instance
I20260810 03:26:19.059849 28292 external_mini_cluster.cc:1428] Setting key 34d43258bdbf2443944e6dfe33a4cb4a
Aug 10 03:26:19 dist-test-slave-vpvm krb5kdc[1971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332378, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.161.3@KRBTEST.COM for kudu/127.27.161.62@KRBTEST.COM
I20260810 03:26:19.076759 2733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:19.056094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.161.3:35849 (local address 127.27.161.62:33451)
0810 03:26:19.056552 (+ 458us) server_negotiation.cc:206] Beginning negotiation
0810 03:26:19.056557 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:26:19.057501 (+ 944us) server_negotiation.cc:407] Connection header received
0810 03:26:19.059674 (+ 2173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:26:19.059681 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:26:19.059777 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:26:19.059879 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:26:19.061613 (+ 1734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:19.062513 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:19.064626 (+ 2113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:19.065069 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:19.065191 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:19.069704 (+ 4513us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:26:19.069711 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:26:19.069715 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:26:19.069757 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:26:19.073026 (+ 3269us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:26:19.074056 (+ 1030us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:26:19.074063 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:26:19.074067 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:26:19.074246 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:26:19.074766 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:26:19.074771 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:26:19.074773 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:26:19.075120 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:26:19.075289 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0810 03:26:19.076261 (+ 972us) server_negotiation.cc:299] Negotiation successful
0810 03:26:19.076454 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":233,"threads_started":1}
I20260810 03:26:19.077804 2734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:19.056594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:33451 (local address 127.27.161.3:35849)
0810 03:26:19.057325 (+ 731us) negotiation.cc:107] Waiting for socket to connect
0810 03:26:19.057367 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0810 03:26:19.059299 (+ 1932us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:26:19.060149 (+ 850us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:26:19.060158 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:26:19.060730 (+ 572us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:26:19.061412 (+ 682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:26:19.061436 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:26:19.062671 (+ 1235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:26:19.062675 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:26:19.064433 (+ 1758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:26:19.064449 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:26:19.065329 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:26:19.065335 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:26:19.065591 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:19.066326 (+ 735us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:26:19.066354 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:26:19.069401 (+ 3047us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:26:19.073255 (+ 3854us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:26:19.073265 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:26:19.073282 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:26:19.073846 (+ 564us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:26:19.074387 (+ 541us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:26:19.074391 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:26:19.074394 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:26:19.074648 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:26:19.075287 (+ 639us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:26:19.075296 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:26:19.075787 (+ 491us) client_negotiation.cc:769] Sending connection context
0810 03:26:19.076177 (+ 390us) client_negotiation.cc:240] Negotiation successful
0810 03:26:19.076647 (+ 470us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":291,"threads_started":1}
I20260810 03:26:19.079380 2731 heartbeater.cc:347] Connected to a master server at 127.27.161.62:33451
I20260810 03:26:19.079741 2731 heartbeater.cc:464] Registering TS with master...
I20260810 03:26:19.080503 2731 heartbeater.cc:511] Master 127.27.161.62:33451 requested a full tablet report, sending...
I20260810 03:26:19.082135 2047 ts_manager.cc:194] Registered new tserver with Master: 17729f053c294195a3f6c51ee83e12cc (127.27.161.3:46821)
I20260810 03:26:19.083019 2047 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.161.3@KRBTEST.COM'} at 127.27.161.3:35849
I20260810 03:26:19.089205 28292 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 03:26:19.109171 2733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:19.094464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54760 (local address 127.27.161.62:33451)
0810 03:26:19.094686 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0810 03:26:19.094691 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:26:19.095285 (+ 594us) server_negotiation.cc:407] Connection header received
0810 03:26:19.095446 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:26:19.095451 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:26:19.095525 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:26:19.095628 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:26:19.097329 (+ 1701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:19.098200 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:19.099564 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:19.100030 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:19.100141 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:19.102701 (+ 2560us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:26:19.102707 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:26:19.102711 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:26:19.102754 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:26:19.105205 (+ 2451us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:26:19.106584 (+ 1379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:26:19.106592 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:26:19.106594 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:26:19.106982 (+ 388us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:26:19.107607 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:26:19.107613 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:26:19.107615 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:26:19.108055 (+ 440us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:26:19.108269 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0810 03:26:19.108695 (+ 426us) server_negotiation.cc:299] Negotiation successful
0810 03:26:19.108925 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260810 03:26:19.114552 2047 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54760:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 03:26:19.121879 2047 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 03:26:19.146404 2750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:19.135456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.2:45177 (local address 127.0.0.1:58046)
0810 03:26:19.136855 (+ 1399us) negotiation.cc:107] Waiting for socket to connect
0810 03:26:19.136912 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0810 03:26:19.137096 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:26:19.138802 (+ 1706us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:26:19.138810 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:26:19.138841 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:26:19.139268 (+ 427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:26:19.139282 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:26:19.141425 (+ 2143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:26:19.141429 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:26:19.143486 (+ 2057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:26:19.143500 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:26:19.145432 (+ 1932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:26:19.145440 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:26:19.145588 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:19.145643 (+ 55us) client_negotiation.cc:769] Sending connection context
0810 03:26:19.145765 (+ 122us) client_negotiation.cc:240] Negotiation successful
0810 03:26:19.145977 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1110,"mutex_wait_us":1,"thread_start_us":2642,"threads_started":1}
I20260810 03:26:19.147576 2747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:19.135458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58046 (local address 127.27.161.2:45177)
0810 03:26:19.137735 (+ 2277us) server_negotiation.cc:206] Beginning negotiation
0810 03:26:19.137745 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 03:26:19.137778 (+ 33us) server_negotiation.cc:407] Connection header received
0810 03:26:19.137887 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:26:19.137895 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:26:19.138552 (+ 657us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:26:19.139473 (+ 921us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:26:19.139708 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:19.141280 (+ 1572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:19.143650 (+ 2370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:19.145147 (+ 1497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:19.145418 (+ 271us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:19.145618 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0810 03:26:19.145835 (+ 217us) server_negotiation.cc:299] Negotiation successful
0810 03:26:19.146054 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2042,"thread_start_us":1929,"threads_started":1}
I20260810 03:26:19.147624 2749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:19.135888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.3:46821 (local address 127.0.0.1:48998)
0810 03:26:19.138405 (+ 2517us) negotiation.cc:107] Waiting for socket to connect
0810 03:26:19.138423 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 03:26:19.138544 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:26:19.139470 (+ 926us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:26:19.139474 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:26:19.139498 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:26:19.139878 (+ 380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:26:19.139892 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:26:19.141737 (+ 1845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:26:19.141741 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:26:19.143897 (+ 2156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:26:19.143910 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:26:19.145249 (+ 1339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:26:19.145256 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:26:19.146998 (+ 1742us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:19.147038 (+ 40us) client_negotiation.cc:769] Sending connection context
0810 03:26:19.147132 (+ 94us) client_negotiation.cc:240] Negotiation successful
0810 03:26:19.147275 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2384,"thread_start_us":895,"threads_started":1}
I20260810 03:26:19.148447 2752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:19.136490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48998 (local address 127.27.161.3:46821)
0810 03:26:19.138182 (+ 1692us) server_negotiation.cc:206] Beginning negotiation
0810 03:26:19.138190 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 03:26:19.138675 (+ 485us) server_negotiation.cc:407] Connection header received
0810 03:26:19.138781 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:26:19.138788 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:26:19.138995 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:26:19.139196 (+ 201us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:26:19.140129 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:19.141605 (+ 1476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:19.144049 (+ 2444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:19.145028 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:19.145167 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:19.145434 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0810 03:26:19.147915 (+ 2481us) server_negotiation.cc:299] Negotiation successful
0810 03:26:19.148112 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1507,"thread_start_us":1283,"threads_started":1}
I20260810 03:26:19.149051 2756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:19.138844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36274 (local address 127.27.161.1:44809)
0810 03:26:19.139832 (+ 988us) server_negotiation.cc:206] Beginning negotiation
0810 03:26:19.139839 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:26:19.140373 (+ 534us) server_negotiation.cc:407] Connection header received
0810 03:26:19.140541 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:26:19.140546 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:26:19.140707 (+ 161us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:26:19.140864 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:26:19.142072 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:19.143654 (+ 1582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:19.145938 (+ 2284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:19.147471 (+ 1533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:19.147606 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:19.148005 (+ 399us) server_negotiation.cc:1035] Waiting for connection context
0810 03:26:19.148651 (+ 646us) server_negotiation.cc:299] Negotiation successful
0810 03:26:19.148807 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":773,"spinlock_wait_cycles":23296,"thread_start_us":743,"threads_started":1}
I20260810 03:26:19.150066 2757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:19.139407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.1:44809 (local address 127.0.0.1:36274)
0810 03:26:19.140255 (+ 848us) negotiation.cc:107] Waiting for socket to connect
0810 03:26:19.140270 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 03:26:19.140388 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:26:19.140878 (+ 490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:26:19.140882 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:26:19.140907 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:26:19.141266 (+ 359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:26:19.141275 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:26:19.143789 (+ 2514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:26:19.143794 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:26:19.145758 (+ 1964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:26:19.145774 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:26:19.148149 (+ 2375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:26:19.148155 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:26:19.148334 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:19.148366 (+ 32us) client_negotiation.cc:769] Sending connection context
0810 03:26:19.148467 (+ 101us) client_negotiation.cc:240] Negotiation successful
0810 03:26:19.149724 (+ 1257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":749,"thread_start_us":621,"threads_started":1}
I20260810 03:26:19.151708 2422 tablet_service.cc:1467] Processing CreateTablet for tablet 35af17d1963846f1a7f2bb67a94c3421 (DEFAULT_TABLE table=test-table [id=bda1b49cd3b44eed823cde80cad53004]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:26:19.152228 2626 tablet_service.cc:1467] Processing CreateTablet for tablet 35af17d1963846f1a7f2bb67a94c3421 (DEFAULT_TABLE table=test-table [id=bda1b49cd3b44eed823cde80cad53004]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:26:19.153450 2422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35af17d1963846f1a7f2bb67a94c3421. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:26:19.153589 2626 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35af17d1963846f1a7f2bb67a94c3421. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:26:19.153846 2229 tablet_service.cc:1467] Processing CreateTablet for tablet 35af17d1963846f1a7f2bb67a94c3421 (DEFAULT_TABLE table=test-table [id=bda1b49cd3b44eed823cde80cad53004]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:26:19.155347 2229 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35af17d1963846f1a7f2bb67a94c3421. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:26:19.167829 2758 tablet_bootstrap.cc:492] T 35af17d1963846f1a7f2bb67a94c3421 P 17729f053c294195a3f6c51ee83e12cc: Bootstrap starting.
I20260810 03:26:19.171317 2758 tablet_bootstrap.cc:654] T 35af17d1963846f1a7f2bb67a94c3421 P 17729f053c294195a3f6c51ee83e12cc: Neither blocks nor log segments found. Creating new log.
I20260810 03:26:19.171939 2760 tablet_bootstrap.cc:492] T 35af17d1963846f1a7f2bb67a94c3421 P e5c76f8b11ba464d9c9d2c878fe4119a: Bootstrap starting.
I20260810 03:26:19.173070 2759 tablet_bootstrap.cc:492] T 35af17d1963846f1a7f2bb67a94c3421 P a93b5c66a3de4c0f995bec49133af2f5: Bootstrap starting.
I20260810 03:26:19.173576 2758 log.cc:824] T 35af17d1963846f1a7f2bb67a94c3421 P 17729f053c294195a3f6c51ee83e12cc: Log is configured to *not* fsync() on all Append() calls
I20260810 03:26:19.176026 2760 tablet_bootstrap.cc:654] T 35af17d1963846f1a7f2bb67a94c3421 P e5c76f8b11ba464d9c9d2c878fe4119a: Neither blocks nor log segments found. Creating new log.
I20260810 03:26:19.176820 2759 tablet_bootstrap.cc:654] T 35af17d1963846f1a7f2bb67a94c3421 P a93b5c66a3de4c0f995bec49133af2f5: Neither blocks nor log segments found. Creating new log.
I20260810 03:26:19.177242 2760 log.cc:824] T 35af17d1963846f1a7f2bb67a94c3421 P e5c76f8b11ba464d9c9d2c878fe4119a: Log is configured to *not* fsync() on all Append() calls
I20260810 03:26:19.177536 2758 tablet_bootstrap.cc:492] T 35af17d1963846f1a7f2bb67a94c3421 P 17729f053c294195a3f6c51ee83e12cc: No bootstrap required, opened a new log
I20260810 03:26:19.177855 2758 ts_tablet_manager.cc:1397] T 35af17d1963846f1a7f2bb67a94c3421 P 17729f053c294195a3f6c51ee83e12cc: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260810 03:26:19.178393 2759 log.cc:824] T 35af17d1963846f1a7f2bb67a94c3421 P a93b5c66a3de4c0f995bec49133af2f5: Log is configured to *not* fsync() on all Append() calls
I20260810 03:26:19.180276 2760 tablet_bootstrap.cc:492] T 35af17d1963846f1a7f2bb67a94c3421 P e5c76f8b11ba464d9c9d2c878fe4119a: No bootstrap required, opened a new log
I20260810 03:26:19.180542 2760 ts_tablet_manager.cc:1397] T 35af17d1963846f1a7f2bb67a94c3421 P e5c76f8b11ba464d9c9d2c878fe4119a: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.006s
I20260810 03:26:19.182076 2759 tablet_bootstrap.cc:492] T 35af17d1963846f1a7f2bb67a94c3421 P a93b5c66a3de4c0f995bec49133af2f5: No bootstrap required, opened a new log
I20260810 03:26:19.182425 2759 ts_tablet_manager.cc:1397] T 35af17d1963846f1a7f2bb67a94c3421 P a93b5c66a3de4c0f995bec49133af2f5: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20260810 03:26:19.188704 2760 raft_consensus.cc:348] T 35af17d1963846f1a7f2bb67a94c3421 P e5c76f8b11ba464d9c9d2c878fe4119a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a93b5c66a3de4c0f995bec49133af2f5" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 45177 } } peers { permanent_uuid: "17729f053c294195a3f6c51ee83e12cc" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 46821 } } peers { permanent_uuid: "e5c76f8b11ba464d9c9d2c878fe4119a" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 44809 } }
I20260810 03:26:19.189137 2760 raft_consensus.cc:374] T 35af17d1963846f1a7f2bb67a94c3421 P e5c76f8b11ba464d9c9d2c878fe4119a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:26:19.189262 2760 raft_consensus.cc:729] T 35af17d1963846f1a7f2bb67a94c3421 P e5c76f8b11ba464d9c9d2c878fe4119a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5c76f8b11ba464d9c9d2c878fe4119a, State: Initialized, Role: FOLLOWER
I20260810 03:26:19.189291 2758 raft_consensus.cc:348] T 35af17d1963846f1a7f2bb67a94c3421 P 17729f053c294195a3f6c51ee83e12cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a93b5c66a3de4c0f995bec49133af2f5" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 45177 } } peers { permanent_uuid: "17729f053c294195a3f6c51ee83e12cc" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 46821 } } peers { permanent_uuid: "e5c76f8b11ba464d9c9d2c878fe4119a" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 44809 } }
I20260810 03:26:19.189786 2758 raft_consensus.cc:374] T 35af17d1963846f1a7f2bb67a94c3421 P 17729f053c294195a3f6c51ee83e12cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:26:19.189898 2758 raft_consensus.cc:729] T 35af17d1963846f1a7f2bb67a94c3421 P 17729f053c294195a3f6c51ee83e12cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17729f053c294195a3f6c51ee83e12cc, State: Initialized, Role: FOLLOWER
I20260810 03:26:19.189941 2760 consensus_queue.cc:260] T 35af17d1963846f1a7f2bb67a94c3421 P e5c76f8b11ba464d9c9d2c878fe4119a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a93b5c66a3de4c0f995bec49133af2f5" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 45177 } } peers { permanent_uuid: "17729f053c294195a3f6c51ee83e12cc" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 46821 } } peers { permanent_uuid: "e5c76f8b11ba464d9c9d2c878fe4119a" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 44809 } }
I20260810 03:26:19.190495 2758 consensus_queue.cc:260] T 35af17d1963846f1a7f2bb67a94c3421 P 17729f053c294195a3f6c51ee83e12cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a93b5c66a3de4c0f995bec49133af2f5" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 45177 } } peers { permanent_uuid: "17729f053c294195a3f6c51ee83e12cc" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 46821 } } peers { permanent_uuid: "e5c76f8b11ba464d9c9d2c878fe4119a" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 44809 } }
I20260810 03:26:19.193423 2760 ts_tablet_manager.cc:1428] T 35af17d1963846f1a7f2bb67a94c3421 P e5c76f8b11ba464d9c9d2c878fe4119a: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
I20260810 03:26:19.193239 2759 raft_consensus.cc:348] T 35af17d1963846f1a7f2bb67a94c3421 P a93b5c66a3de4c0f995bec49133af2f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a93b5c66a3de4c0f995bec49133af2f5" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 45177 } } peers { permanent_uuid: "17729f053c294195a3f6c51ee83e12cc" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 46821 } } peers { permanent_uuid: "e5c76f8b11ba464d9c9d2c878fe4119a" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 44809 } }
I20260810 03:26:19.193789 2759 raft_consensus.cc:374] T 35af17d1963846f1a7f2bb67a94c3421 P a93b5c66a3de4c0f995bec49133af2f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:26:19.193919 2759 raft_consensus.cc:729] T 35af17d1963846f1a7f2bb67a94c3421 P a93b5c66a3de4c0f995bec49133af2f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a93b5c66a3de4c0f995bec49133af2f5, State: Initialized, Role: FOLLOWER
I20260810 03:26:19.194581 2731 heartbeater.cc:503] Master 127.27.161.62:33451 was elected leader, sending a full tablet report...
I20260810 03:26:19.194777 2759 consensus_queue.cc:260] T 35af17d1963846f1a7f2bb67a94c3421 P a93b5c66a3de4c0f995bec49133af2f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a93b5c66a3de4c0f995bec49133af2f5" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 45177 } } peers { permanent_uuid: "17729f053c294195a3f6c51ee83e12cc" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 46821 } } peers { permanent_uuid: "e5c76f8b11ba464d9c9d2c878fe4119a" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 44809 } }
I20260810 03:26:19.195230 2758 ts_tablet_manager.cc:1428] T 35af17d1963846f1a7f2bb67a94c3421 P 17729f053c294195a3f6c51ee83e12cc: Time spent starting tablet: real 0.017s user 0.013s sys 0.003s
I20260810 03:26:19.196346 2529 heartbeater.cc:503] Master 127.27.161.62:33451 was elected leader, sending a full tablet report...
I20260810 03:26:19.196933 2759 ts_tablet_manager.cc:1428] T 35af17d1963846f1a7f2bb67a94c3421 P a93b5c66a3de4c0f995bec49133af2f5: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260810 03:26:19.198611 2764 raft_consensus.cc:482] T 35af17d1963846f1a7f2bb67a94c3421 P e5c76f8b11ba464d9c9d2c878fe4119a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 03:26:19.198872 2764 raft_consensus.cc:504] T 35af17d1963846f1a7f2bb67a94c3421 P e5c76f8b11ba464d9c9d2c878fe4119a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a93b5c66a3de4c0f995bec49133af2f5" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 45177 } } peers { permanent_uuid: "17729f053c294195a3f6c51ee83e12cc" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 46821 } } peers { permanent_uuid: "e5c76f8b11ba464d9c9d2c878fe4119a" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 44809 } }
I20260810 03:26:19.200760 2764 leader_election.cc:288] T 35af17d1963846f1a7f2bb67a94c3421 P e5c76f8b11ba464d9c9d2c878fe4119a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a93b5c66a3de4c0f995bec49133af2f5 (127.27.161.2:45177), 17729f053c294195a3f6c51ee83e12cc (127.27.161.3:46821)
W20260810 03:26:19.205230 2324 tablet.cc:2369] T 35af17d1963846f1a7f2bb67a94c3421 P e5c76f8b11ba464d9c9d2c878fe4119a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 03:26:19.209657 2752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:19.201141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.161.1:55921 (local address 127.27.161.3:46821)
0810 03:26:19.201339 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0810 03:26:19.201344 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:26:19.202945 (+ 1601us) server_negotiation.cc:407] Connection header received
0810 03:26:19.203126 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:26:19.203132 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:26:19.203213 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:26:19.203322 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:26:19.203947 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:19.204901 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:19.207739 (+ 2838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:19.208621 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:19.208772 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:19.208834 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0810 03:26:19.209244 (+ 410us) server_negotiation.cc:299] Negotiation successful
0810 03:26:19.209387 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260810 03:26:19.210175 2769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:19.201941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.2:45177 (local address 127.27.161.1:51181)
0810 03:26:19.203554 (+ 1613us) negotiation.cc:107] Waiting for socket to connect
0810 03:26:19.203571 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 03:26:19.203678 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:26:19.203971 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:26:19.203975 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:26:19.204001 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:26:19.204368 (+ 367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:26:19.204381 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:26:19.206102 (+ 1721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:26:19.206106 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:26:19.207788 (+ 1682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:26:19.207804 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:26:19.209264 (+ 1460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:26:19.209270 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:26:19.209437 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:19.209470 (+ 33us) client_negotiation.cc:769] Sending connection context
0810 03:26:19.209571 (+ 101us) client_negotiation.cc:240] Negotiation successful
0810 03:26:19.209700 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1449,"mutex_wait_us":25,"thread_start_us":1237,"threads_started":1}
I20260810 03:26:19.210206 2768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:19.201922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.3:46821 (local address 127.27.161.1:55921)
0810 03:26:19.202820 (+ 898us) negotiation.cc:107] Waiting for socket to connect
0810 03:26:19.202838 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 03:26:19.202990 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:26:19.203373 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:26:19.203382 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:26:19.203414 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:26:19.203784 (+ 370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:26:19.203798 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:26:19.205122 (+ 1324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:26:19.205136 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:26:19.207536 (+ 2400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:26:19.207559 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:26:19.208870 (+ 1311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:26:19.208877 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:26:19.209071 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:19.209112 (+ 41us) client_negotiation.cc:769] Sending connection context
0810 03:26:19.209216 (+ 104us) client_negotiation.cc:240] Negotiation successful
0810 03:26:19.209384 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":775,"mutex_wait_us":26,"thread_start_us":415,"threads_started":1}
I20260810 03:26:19.210489 2747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:19.201465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.161.1:51181 (local address 127.27.161.2:45177)
0810 03:26:19.201640 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0810 03:26:19.201646 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:26:19.203659 (+ 2013us) server_negotiation.cc:407] Connection header received
0810 03:26:19.203780 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:26:19.203785 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:26:19.203860 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:26:19.203943 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:26:19.204661 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:19.205870 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:19.208023 (+ 2153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:19.209014 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:19.209198 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:19.209270 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0810 03:26:19.209695 (+ 425us) server_negotiation.cc:299] Negotiation successful
0810 03:26:19.210227 (+ 532us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260810 03:26:19.211198 2657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35af17d1963846f1a7f2bb67a94c3421" candidate_uuid: "e5c76f8b11ba464d9c9d2c878fe4119a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17729f053c294195a3f6c51ee83e12cc" is_pre_election: true
I20260810 03:26:19.211508 2453 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35af17d1963846f1a7f2bb67a94c3421" candidate_uuid: "e5c76f8b11ba464d9c9d2c878fe4119a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a93b5c66a3de4c0f995bec49133af2f5" is_pre_election: true
I20260810 03:26:19.211743 2657 raft_consensus.cc:2454] T 35af17d1963846f1a7f2bb67a94c3421 P 17729f053c294195a3f6c51ee83e12cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5c76f8b11ba464d9c9d2c878fe4119a in term 0.
I20260810 03:26:19.211977 2453 raft_consensus.cc:2454] T 35af17d1963846f1a7f2bb67a94c3421 P a93b5c66a3de4c0f995bec49133af2f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5c76f8b11ba464d9c9d2c878fe4119a in term 0.
I20260810 03:26:19.212423 2159 leader_election.cc:302] T 35af17d1963846f1a7f2bb67a94c3421 P e5c76f8b11ba464d9c9d2c878fe4119a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17729f053c294195a3f6c51ee83e12cc, e5c76f8b11ba464d9c9d2c878fe4119a; no voters:
I20260810 03:26:19.212793 2764 raft_consensus.cc:2781] T 35af17d1963846f1a7f2bb67a94c3421 P e5c76f8b11ba464d9c9d2c878fe4119a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 03:26:19.212940 2764 raft_consensus.cc:482] T 35af17d1963846f1a7f2bb67a94c3421 P e5c76f8b11ba464d9c9d2c878fe4119a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 03:26:19.213032 2764 raft_consensus.cc:3037] T 35af17d1963846f1a7f2bb67a94c3421 P e5c76f8b11ba464d9c9d2c878fe4119a [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:26:19.215850 2764 raft_consensus.cc:504] T 35af17d1963846f1a7f2bb67a94c3421 P e5c76f8b11ba464d9c9d2c878fe4119a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a93b5c66a3de4c0f995bec49133af2f5" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 45177 } } peers { permanent_uuid: "17729f053c294195a3f6c51ee83e12cc" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 46821 } } peers { permanent_uuid: "e5c76f8b11ba464d9c9d2c878fe4119a" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 44809 } }
I20260810 03:26:19.216549 2764 leader_election.cc:288] T 35af17d1963846f1a7f2bb67a94c3421 P e5c76f8b11ba464d9c9d2c878fe4119a [CANDIDATE]: Term 1 election: Requested vote from peers a93b5c66a3de4c0f995bec49133af2f5 (127.27.161.2:45177), 17729f053c294195a3f6c51ee83e12cc (127.27.161.3:46821)
I20260810 03:26:19.217103 2453 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35af17d1963846f1a7f2bb67a94c3421" candidate_uuid: "e5c76f8b11ba464d9c9d2c878fe4119a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a93b5c66a3de4c0f995bec49133af2f5"
I20260810 03:26:19.217145 2657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35af17d1963846f1a7f2bb67a94c3421" candidate_uuid: "e5c76f8b11ba464d9c9d2c878fe4119a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17729f053c294195a3f6c51ee83e12cc"
I20260810 03:26:19.217358 2657 raft_consensus.cc:3037] T 35af17d1963846f1a7f2bb67a94c3421 P 17729f053c294195a3f6c51ee83e12cc [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:26:19.217343 2453 raft_consensus.cc:3037] T 35af17d1963846f1a7f2bb67a94c3421 P a93b5c66a3de4c0f995bec49133af2f5 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:26:19.220671 2657 raft_consensus.cc:2454] T 35af17d1963846f1a7f2bb67a94c3421 P 17729f053c294195a3f6c51ee83e12cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e5c76f8b11ba464d9c9d2c878fe4119a in term 1.
I20260810 03:26:19.220671 2453 raft_consensus.cc:2454] T 35af17d1963846f1a7f2bb67a94c3421 P a93b5c66a3de4c0f995bec49133af2f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e5c76f8b11ba464d9c9d2c878fe4119a in term 1.
I20260810 03:26:19.221180 2160 leader_election.cc:302] T 35af17d1963846f1a7f2bb67a94c3421 P e5c76f8b11ba464d9c9d2c878fe4119a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 17729f053c294195a3f6c51ee83e12cc, a93b5c66a3de4c0f995bec49133af2f5, e5c76f8b11ba464d9c9d2c878fe4119a; no voters:
I20260810 03:26:19.221515 2764 raft_consensus.cc:2781] T 35af17d1963846f1a7f2bb67a94c3421 P e5c76f8b11ba464d9c9d2c878fe4119a [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:26:19.222030 2764 raft_consensus.cc:686] T 35af17d1963846f1a7f2bb67a94c3421 P e5c76f8b11ba464d9c9d2c878fe4119a [term 1 LEADER]: Becoming Leader. State: Replica: e5c76f8b11ba464d9c9d2c878fe4119a, State: Running, Role: LEADER
I20260810 03:26:19.222602 2764 consensus_queue.cc:237] T 35af17d1963846f1a7f2bb67a94c3421 P e5c76f8b11ba464d9c9d2c878fe4119a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a93b5c66a3de4c0f995bec49133af2f5" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 45177 } } peers { permanent_uuid: "17729f053c294195a3f6c51ee83e12cc" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 46821 } } peers { permanent_uuid: "e5c76f8b11ba464d9c9d2c878fe4119a" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 44809 } }
I20260810 03:26:19.227710 2046 catalog_manager.cc:5495] T 35af17d1963846f1a7f2bb67a94c3421 P e5c76f8b11ba464d9c9d2c878fe4119a reported cstate change: term changed from 0 to 1, leader changed from <none> to e5c76f8b11ba464d9c9d2c878fe4119a (127.27.161.1). New cstate: current_term: 1 leader_uuid: "e5c76f8b11ba464d9c9d2c878fe4119a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a93b5c66a3de4c0f995bec49133af2f5" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 45177 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "17729f053c294195a3f6c51ee83e12cc" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 46821 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e5c76f8b11ba464d9c9d2c878fe4119a" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 44809 } health_report { overall_health: HEALTHY } } }
I20260810 03:26:19.250597 2756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:19.245085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36286 (local address 127.27.161.1:44809)
0810 03:26:19.245304 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0810 03:26:19.245311 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:26:19.245376 (+ 65us) server_negotiation.cc:407] Connection header received
0810 03:26:19.245651 (+ 275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:26:19.245655 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:26:19.245748 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:26:19.245888 (+ 140us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 03:26:19.246316 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:19.247038 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:19.248418 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:19.248828 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:19.248943 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:19.249444 (+ 501us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 03:26:19.250007 (+ 563us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 03:26:19.250154 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0810 03:26:19.250243 (+ 89us) server_negotiation.cc:299] Negotiation successful
0810 03:26:19.250332 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
W20260810 03:26:19.251912 2530 tablet.cc:2369] T 35af17d1963846f1a7f2bb67a94c3421 P a93b5c66a3de4c0f995bec49133af2f5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 03:26:19.259900 2657 raft_consensus.cc:1261] T 35af17d1963846f1a7f2bb67a94c3421 P 17729f053c294195a3f6c51ee83e12cc [term 1 FOLLOWER]: Refusing update from remote peer e5c76f8b11ba464d9c9d2c878fe4119a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:26:19.259900 2453 raft_consensus.cc:1261] T 35af17d1963846f1a7f2bb67a94c3421 P a93b5c66a3de4c0f995bec49133af2f5 [term 1 FOLLOWER]: Refusing update from remote peer e5c76f8b11ba464d9c9d2c878fe4119a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:26:19.261128 2771 consensus_queue.cc:1035] T 35af17d1963846f1a7f2bb67a94c3421 P e5c76f8b11ba464d9c9d2c878fe4119a [LEADER]: Connected to new peer: Peer: permanent_uuid: "17729f053c294195a3f6c51ee83e12cc" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 46821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:26:19.261493 2764 consensus_queue.cc:1035] T 35af17d1963846f1a7f2bb67a94c3421 P e5c76f8b11ba464d9c9d2c878fe4119a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a93b5c66a3de4c0f995bec49133af2f5" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 45177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:26:19.274915 2774 mvcc.cc:205] Tried to move back new op lower bound from 7316817425436930048 to 7316817425299750912. Current Snapshot: MvccSnapshot[applied={T|T < 7316817425436930048}]
I20260810 03:26:19.291786 2046 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54760:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 03:26:19.292047 2046 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54760:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 03:26:19.296072 2046 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 79ba9581131e4f2ba9d01474d4d54360: Sending DeleteTablet for 3 replicas of tablet 35af17d1963846f1a7f2bb67a94c3421
I20260810 03:26:19.297523 2626 tablet_service.cc:1514] Processing DeleteTablet for tablet 35af17d1963846f1a7f2bb67a94c3421 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:26:19 UTC) from {username='kudu', principal='kudu/127.27.161.62@KRBTEST.COM'} at 127.0.0.1:48998
I20260810 03:26:19.297676 2422 tablet_service.cc:1514] Processing DeleteTablet for tablet 35af17d1963846f1a7f2bb67a94c3421 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:26:19 UTC) from {username='kudu', principal='kudu/127.27.161.62@KRBTEST.COM'} at 127.0.0.1:58046
I20260810 03:26:19.297829 2229 tablet_service.cc:1514] Processing DeleteTablet for tablet 35af17d1963846f1a7f2bb67a94c3421 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:26:19 UTC) from {username='kudu', principal='kudu/127.27.161.62@KRBTEST.COM'} at 127.0.0.1:36274
I20260810 03:26:19.300172 28292 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iBH5/build/asan/bin/kudu pid 2135
I20260810 03:26:19.308097 2790 tablet_replica.cc:330] T 35af17d1963846f1a7f2bb67a94c3421 P e5c76f8b11ba464d9c9d2c878fe4119a: stopping tablet replica
I20260810 03:26:19.308863 2790 raft_consensus.cc:2229] T 35af17d1963846f1a7f2bb67a94c3421 P e5c76f8b11ba464d9c9d2c878fe4119a [term 1 LEADER]: Raft consensus shutting down.
I20260810 03:26:19.309721 2790 raft_consensus.cc:2258] T 35af17d1963846f1a7f2bb67a94c3421 P e5c76f8b11ba464d9c9d2c878fe4119a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:26:19.310720 2788 tablet_replica.cc:330] T 35af17d1963846f1a7f2bb67a94c3421 P 17729f053c294195a3f6c51ee83e12cc: stopping tablet replica
I20260810 03:26:19.311117 2788 raft_consensus.cc:2229] T 35af17d1963846f1a7f2bb67a94c3421 P 17729f053c294195a3f6c51ee83e12cc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 03:26:19.311573 2788 raft_consensus.cc:2258] T 35af17d1963846f1a7f2bb67a94c3421 P 17729f053c294195a3f6c51ee83e12cc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:26:19.313750 2789 tablet_replica.cc:330] T 35af17d1963846f1a7f2bb67a94c3421 P a93b5c66a3de4c0f995bec49133af2f5: stopping tablet replica
I20260810 03:26:19.314090 2788 ts_tablet_manager.cc:1905] T 35af17d1963846f1a7f2bb67a94c3421 P 17729f053c294195a3f6c51ee83e12cc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:26:19.314328 2789 raft_consensus.cc:2229] T 35af17d1963846f1a7f2bb67a94c3421 P a93b5c66a3de4c0f995bec49133af2f5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 03:26:19.314953 2789 raft_consensus.cc:2258] T 35af17d1963846f1a7f2bb67a94c3421 P a93b5c66a3de4c0f995bec49133af2f5 [term 1 FOLLOWER]: Raft consensus is shut down!
Aug 10 03:26:19 dist-test-slave-vpvm krb5kdc[1971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332376, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.161.1@KRBTEST.COM
I20260810 03:26:19.323992 2788 ts_tablet_manager.cc:1918] T 35af17d1963846f1a7f2bb67a94c3421 P 17729f053c294195a3f6c51ee83e12cc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 03:26:19.324204 2788 log.cc:1196] T 35af17d1963846f1a7f2bb67a94c3421 P 17729f053c294195a3f6c51ee83e12cc: Deleting WAL directory at /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786332336319544-28292-0/minicluster-data/ts-2/wal/wals/35af17d1963846f1a7f2bb67a94c3421
I20260810 03:26:19.325233 2790 ts_tablet_manager.cc:1905] T 35af17d1963846f1a7f2bb67a94c3421 P e5c76f8b11ba464d9c9d2c878fe4119a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:26:19.328148 2789 ts_tablet_manager.cc:1905] T 35af17d1963846f1a7f2bb67a94c3421 P a93b5c66a3de4c0f995bec49133af2f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:26:19.333244 2790 ts_tablet_manager.cc:1918] T 35af17d1963846f1a7f2bb67a94c3421 P e5c76f8b11ba464d9c9d2c878fe4119a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 03:26:19.334120 2788 ts_tablet_manager.cc:1939] T 35af17d1963846f1a7f2bb67a94c3421 P 17729f053c294195a3f6c51ee83e12cc: Deleting consensus metadata
I20260810 03:26:19.333642 2756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:19.301834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36290 (local address 127.27.161.1:44809)
0810 03:26:19.302056 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0810 03:26:19.302062 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:26:19.313042 (+ 10980us) server_negotiation.cc:407] Connection header received
0810 03:26:19.313245 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:26:19.313252 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:26:19.313344 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:26:19.313449 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:26:19.315413 (+ 1964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:19.316577 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:19.318753 (+ 2176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:19.319472 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:19.319727 (+ 255us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:19.325481 (+ 5754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:26:19.325496 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:26:19.325516 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:26:19.325566 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:26:19.329686 (+ 4120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:26:19.330751 (+ 1065us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:26:19.330765 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:26:19.330769 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:26:19.330970 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:26:19.331453 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:26:19.331458 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:26:19.331462 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:26:19.332005 (+ 543us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:26:19.332339 (+ 334us) server_negotiation.cc:1035] Waiting for connection context
0810 03:26:19.332952 (+ 613us) server_negotiation.cc:299] Negotiation successful
0810 03:26:19.333231 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260810 03:26:19.335613 2315 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:26:19.335755 2789 ts_tablet_manager.cc:1918] T 35af17d1963846f1a7f2bb67a94c3421 P a93b5c66a3de4c0f995bec49133af2f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 03:26:19.334403 2790 log.cc:1196] T 35af17d1963846f1a7f2bb67a94c3421 P e5c76f8b11ba464d9c9d2c878fe4119a: Deleting WAL directory at /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786332336319544-28292-0/minicluster-data/ts-0/wal/wals/35af17d1963846f1a7f2bb67a94c3421
I20260810 03:26:19.336608 2022 catalog_manager.cc:4841] TS 17729f053c294195a3f6c51ee83e12cc (127.27.161.3:46821): tablet 35af17d1963846f1a7f2bb67a94c3421 (table test-table [id=bda1b49cd3b44eed823cde80cad53004]) successfully deleted
I20260810 03:26:19.337904 2789 log.cc:1196] T 35af17d1963846f1a7f2bb67a94c3421 P a93b5c66a3de4c0f995bec49133af2f5: Deleting WAL directory at /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786332336319544-28292-0/minicluster-data/ts-1/wal/wals/35af17d1963846f1a7f2bb67a94c3421
I20260810 03:26:19.338567 2789 ts_tablet_manager.cc:1939] T 35af17d1963846f1a7f2bb67a94c3421 P a93b5c66a3de4c0f995bec49133af2f5: Deleting consensus metadata
I20260810 03:26:19.340646 2023 catalog_manager.cc:4841] TS a93b5c66a3de4c0f995bec49133af2f5 (127.27.161.2:45177): tablet 35af17d1963846f1a7f2bb67a94c3421 (table test-table [id=bda1b49cd3b44eed823cde80cad53004]) successfully deleted
I20260810 03:26:19.437201 2790 ts_tablet_manager.cc:1939] T 35af17d1963846f1a7f2bb67a94c3421 P e5c76f8b11ba464d9c9d2c878fe4119a: Deleting consensus metadata
I20260810 03:26:19.437321 28292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iBH5/build/asan/bin/kudu with pid 2135
I20260810 03:26:19.464684 28292 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iBH5/build/asan/bin/kudu pid 2333
W20260810 03:26:19.464797 2023 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.161.1:44809 (error 108)
I20260810 03:26:19.466835 2757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:19.466332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.1:44809 (local address 127.0.0.1:36306)
0810 03:26:19.466513 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0810 03:26:19.466650 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.1:44809: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260810 03:26:19.467121 2023 catalog_manager.cc:4584] TS e5c76f8b11ba464d9c9d2c878fe4119a (127.27.161.1:44809): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 35af17d1963846f1a7f2bb67a94c3421: Network error: Client connection negotiation failed: client connection to 127.27.161.1:44809: connect: Connection refused (error 111)
Aug 10 03:26:19 dist-test-slave-vpvm krb5kdc[1971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332376, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.161.2@KRBTEST.COM
I20260810 03:26:19.480790 2747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:19.465479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58056 (local address 127.27.161.2:45177)
0810 03:26:19.465901 (+ 422us) server_negotiation.cc:206] Beginning negotiation
0810 03:26:19.465905 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:26:19.465925 (+ 20us) server_negotiation.cc:407] Connection header received
0810 03:26:19.466009 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:26:19.466015 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:26:19.466126 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:26:19.466233 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:26:19.467666 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:19.468402 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:19.469747 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:19.470338 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:19.470457 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:19.474960 (+ 4503us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:26:19.474979 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:26:19.474998 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:26:19.475062 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:26:19.478170 (+ 3108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:26:19.478913 (+ 743us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:26:19.478919 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:26:19.478922 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:26:19.479051 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:26:19.479344 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:26:19.479347 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:26:19.479349 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:26:19.479730 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:26:19.479962 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0810 03:26:19.480185 (+ 223us) server_negotiation.cc:299] Negotiation successful
0810 03:26:19.480415 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311}
I20260810 03:26:19.481449 2521 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:26:19.521099 2757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:19.520595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.1:44809 (local address 127.0.0.1:36318)
0810 03:26:19.520839 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0810 03:26:19.520934 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.1:44809: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108}
I20260810 03:26:19.522279 2757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:19.521873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.1:44809 (local address 127.0.0.1:36334)
0810 03:26:19.522049 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0810 03:26:19.522124 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.1:44809: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260810 03:26:19.571252 28292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iBH5/build/asan/bin/kudu with pid 2333
I20260810 03:26:19.582839 2757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:19.582246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.1:44809 (local address 127.0.0.1:36342)
0810 03:26:19.582503 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0810 03:26:19.582617 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.1:44809: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260810 03:26:19.584002 2757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:19.583577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.1:44809 (local address 127.0.0.1:36346)
0810 03:26:19.583750 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0810 03:26:19.583845 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.1:44809: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260810 03:26:19.596629 28292 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iBH5/build/asan/bin/kudu pid 2536
Aug 10 03:26:19 dist-test-slave-vpvm krb5kdc[1971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332376, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.161.3@KRBTEST.COM
I20260810 03:26:19.611825 2752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:19.597416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49004 (local address 127.27.161.3:46821)
0810 03:26:19.597599 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0810 03:26:19.597603 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:26:19.597622 (+ 19us) server_negotiation.cc:407] Connection header received
0810 03:26:19.597747 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:26:19.597751 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:26:19.597828 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:26:19.597912 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:26:19.599180 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:19.599901 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:19.601106 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:19.601499 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:19.601592 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:19.606218 (+ 4626us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:26:19.606236 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:26:19.606256 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:26:19.606305 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:26:19.608960 (+ 2655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:26:19.609781 (+ 821us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:26:19.609789 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:26:19.609791 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:26:19.609936 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:26:19.610316 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:26:19.610319 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:26:19.610321 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:26:19.610705 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:26:19.610941 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0810 03:26:19.611294 (+ 353us) server_negotiation.cc:299] Negotiation successful
0810 03:26:19.611481 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260810 03:26:19.612543 2725 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:26:19.664825 2757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:19.664293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.1:44809 (local address 127.0.0.1:36358)
0810 03:26:19.664560 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0810 03:26:19.664665 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.1:44809: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
I20260810 03:26:19.665915 2757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:19.665475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.1:44809 (local address 127.0.0.1:36366)
0810 03:26:19.665712 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0810 03:26:19.665787 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.1:44809: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135}
I20260810 03:26:19.705298 28292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iBH5/build/asan/bin/kudu with pid 2536
I20260810 03:26:19.731279 28292 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iBH5/build/asan/bin/kudu pid 1989
I20260810 03:26:19.732558 2094 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:26:19.845845 2757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:19.844891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.1:44809 (local address 127.0.0.1:36372)
0810 03:26:19.845429 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0810 03:26:19.845561 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.1:44809: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":350}
I20260810 03:26:19.846494 28292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iBH5/build/asan/bin/kudu with pid 1989
2026-08-10T03:26:19Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (4469 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-08-10T03:26:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T03:26:19Z Disabled control of system clock
I20260810 03:26:19.935442 28292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iBH5/build/asan/bin/kudu
/tmp/dist-test-task24iBH5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786332336319544-28292-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786332336319544-28292-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786332336319544-28292-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786332336319544-28292-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.161.62:42379
--webserver_interface=127.27.161.62
--webserver_port=0
--builtin_ntp_servers=127.27.161.20:36911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.161.62:42379
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260810 03:26:20.291477 2805 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:26:20.291791 2805 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:26:20.291905 2805 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:26:20.299474 2805 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:26:20.299604 2805 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:26:20.299647 2805 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:26:20.299727 2805 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:26:20.299780 2805 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:26:20.309352 2805 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.161.20:36911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786332336319544-28292-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786332336319544-28292-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.161.62:42379
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.161.62:42379
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786332336319544-28292-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.161.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786332336319544-28292-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:26:20.310889 2805 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:26:20.312812 2805 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:26:20.324992 2816 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:26:20.324960 2817 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:26:20.324960 2820 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:26:20.325542 2805 server_base.cc:1034] running on GCE node
I20260810 03:26:20.326504 2805 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:26:20.327957 2805 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:26:20.329190 2805 hybrid_clock.cc:648] HybridClock initialized: now 1786332380329137 us; error 49 us; skew 500 ppm
I20260810 03:26:20.329571 2805 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 03:26:20.334976 2805 webserver.cc:460] Webserver started at http://127.27.161.62:35845/ using document root <none> and password file <none>
I20260810 03:26:20.335803 2805 fs_manager.cc:359] Metadata directory not provided
I20260810 03:26:20.335909 2805 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:26:20.336283 2805 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:26:20.339510 2805 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786332336319544-28292-0/minicluster-data/master-0/data/instance:
uuid: "b85e7fb7855e4c2282d7b0d2b3a71da9"
format_stamp: "Formatted at 2026-08-10 03:26:20 on dist-test-slave-vpvm"
server_key: "f4e54e88669c887d652422dd592a7e50"
server_key_iv: "804825931609e8e3f81ace0d9188cce8"
server_key_version: "encryptionkey@0"
I20260810 03:26:20.340342 2805 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786332336319544-28292-0/minicluster-data/master-0/wal/instance:
uuid: "b85e7fb7855e4c2282d7b0d2b3a71da9"
format_stamp: "Formatted at 2026-08-10 03:26:20 on dist-test-slave-vpvm"
server_key: "f4e54e88669c887d652422dd592a7e50"
server_key_iv: "804825931609e8e3f81ace0d9188cce8"
server_key_version: "encryptionkey@0"
I20260810 03:26:20.346024 2805 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260810 03:26:20.350049 2828 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:26:20.351573 2805 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260810 03:26:20.351783 2805 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786332336319544-28292-0/minicluster-data/master-0/data,/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786332336319544-28292-0/minicluster-data/master-0/wal
uuid: "b85e7fb7855e4c2282d7b0d2b3a71da9"
format_stamp: "Formatted at 2026-08-10 03:26:20 on dist-test-slave-vpvm"
server_key: "f4e54e88669c887d652422dd592a7e50"
server_key_iv: "804825931609e8e3f81ace0d9188cce8"
server_key_version: "encryptionkey@0"
I20260810 03:26:20.352028 2805 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786332336319544-28292-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786332336319544-28292-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786332336319544-28292-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:26:20.358297 2805 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:26:20.359206 2805 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:26:20.359513 2805 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:26:20.377878 2805 rpc_server.cc:307] RPC server started. Bound to: 127.27.161.62:42379
I20260810 03:26:20.377897 2908 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.161.62:42379 every 8 connection(s)
I20260810 03:26:20.379446 2805 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786332336319544-28292-0/minicluster-data/master-0/data/info.pb
I20260810 03:26:20.382020 28292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iBH5/build/asan/bin/kudu as pid 2805
I20260810 03:26:20.382221 28292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786332336319544-28292-0/minicluster-data/master-0/wal/instance
I20260810 03:26:20.383034 28292 external_mini_cluster.cc:1428] Setting key decf64a24cb6a2574f0e08f77300547a
I20260810 03:26:20.383641 2910 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:26:20.393486 2915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:20.385699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43356 (local address 127.27.161.62:42379)
0810 03:26:20.386986 (+ 1287us) server_negotiation.cc:206] Beginning negotiation
0810 03:26:20.387004 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0810 03:26:20.387075 (+ 71us) server_negotiation.cc:407] Connection header received
0810 03:26:20.388950 (+ 1875us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:26:20.388982 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:26:20.389612 (+ 630us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:26:20.390162 (+ 550us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:26:20.391300 (+ 1138us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:26:20.391310 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:26:20.391328 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:26:20.391348 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:26:20.391432 (+ 84us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:26:20.391454 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:26:20.391716 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0810 03:26:20.391872 (+ 156us) server_negotiation.cc:299] Negotiation successful
0810 03:26:20.392198 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":636,"thread_start_us":397,"threads_started":1}
I20260810 03:26:20.399296 2910 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b85e7fb7855e4c2282d7b0d2b3a71da9: Bootstrap starting.
I20260810 03:26:20.402596 2910 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b85e7fb7855e4c2282d7b0d2b3a71da9: Neither blocks nor log segments found. Creating new log.
I20260810 03:26:20.403683 2910 log.cc:824] T 00000000000000000000000000000000 P b85e7fb7855e4c2282d7b0d2b3a71da9: Log is configured to *not* fsync() on all Append() calls
I20260810 03:26:20.406697 2910 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b85e7fb7855e4c2282d7b0d2b3a71da9: No bootstrap required, opened a new log
I20260810 03:26:20.414434 2910 raft_consensus.cc:348] T 00000000000000000000000000000000 P b85e7fb7855e4c2282d7b0d2b3a71da9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b85e7fb7855e4c2282d7b0d2b3a71da9" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 42379 } }
I20260810 03:26:20.414852 2910 raft_consensus.cc:374] T 00000000000000000000000000000000 P b85e7fb7855e4c2282d7b0d2b3a71da9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:26:20.414966 2910 raft_consensus.cc:729] T 00000000000000000000000000000000 P b85e7fb7855e4c2282d7b0d2b3a71da9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b85e7fb7855e4c2282d7b0d2b3a71da9, State: Initialized, Role: FOLLOWER
I20260810 03:26:20.415570 2910 consensus_queue.cc:260] T 00000000000000000000000000000000 P b85e7fb7855e4c2282d7b0d2b3a71da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b85e7fb7855e4c2282d7b0d2b3a71da9" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 42379 } }
I20260810 03:26:20.415786 2910 raft_consensus.cc:388] T 00000000000000000000000000000000 P b85e7fb7855e4c2282d7b0d2b3a71da9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 03:26:20.415921 2910 raft_consensus.cc:482] T 00000000000000000000000000000000 P b85e7fb7855e4c2282d7b0d2b3a71da9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 03:26:20.416085 2910 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b85e7fb7855e4c2282d7b0d2b3a71da9 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:26:20.418433 2910 raft_consensus.cc:504] T 00000000000000000000000000000000 P b85e7fb7855e4c2282d7b0d2b3a71da9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b85e7fb7855e4c2282d7b0d2b3a71da9" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 42379 } }
I20260810 03:26:20.418998 2910 leader_election.cc:302] T 00000000000000000000000000000000 P b85e7fb7855e4c2282d7b0d2b3a71da9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b85e7fb7855e4c2282d7b0d2b3a71da9; no voters:
I20260810 03:26:20.419764 2910 leader_election.cc:288] T 00000000000000000000000000000000 P b85e7fb7855e4c2282d7b0d2b3a71da9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 03:26:20.420149 2918 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b85e7fb7855e4c2282d7b0d2b3a71da9 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:26:20.421280 2918 raft_consensus.cc:686] T 00000000000000000000000000000000 P b85e7fb7855e4c2282d7b0d2b3a71da9 [term 1 LEADER]: Becoming Leader. State: Replica: b85e7fb7855e4c2282d7b0d2b3a71da9, State: Running, Role: LEADER
I20260810 03:26:20.422005 2918 consensus_queue.cc:237] T 00000000000000000000000000000000 P b85e7fb7855e4c2282d7b0d2b3a71da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b85e7fb7855e4c2282d7b0d2b3a71da9" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 42379 } }
I20260810 03:26:20.422564 2910 sys_catalog.cc:564] T 00000000000000000000000000000000 P b85e7fb7855e4c2282d7b0d2b3a71da9 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:26:20.424955 2921 sys_catalog.cc:455] T 00000000000000000000000000000000 P b85e7fb7855e4c2282d7b0d2b3a71da9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b85e7fb7855e4c2282d7b0d2b3a71da9. Latest consensus state: current_term: 1 leader_uuid: "b85e7fb7855e4c2282d7b0d2b3a71da9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b85e7fb7855e4c2282d7b0d2b3a71da9" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 42379 } } }
I20260810 03:26:20.425359 2921 sys_catalog.cc:458] T 00000000000000000000000000000000 P b85e7fb7855e4c2282d7b0d2b3a71da9 [sys.catalog]: This master's current role is: LEADER
I20260810 03:26:20.425498 2919 sys_catalog.cc:455] T 00000000000000000000000000000000 P b85e7fb7855e4c2282d7b0d2b3a71da9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b85e7fb7855e4c2282d7b0d2b3a71da9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b85e7fb7855e4c2282d7b0d2b3a71da9" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 42379 } } }
I20260810 03:26:20.425832 2919 sys_catalog.cc:458] T 00000000000000000000000000000000 P b85e7fb7855e4c2282d7b0d2b3a71da9 [sys.catalog]: This master's current role is: LEADER
I20260810 03:26:20.429435 2925 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 03:26:20.434932 2925 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 03:26:20.444739 2925 catalog_manager.cc:1347] Generated new cluster ID: 33795bc10656423382c2c06efa4f796b
I20260810 03:26:20.444880 2925 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 03:26:20.474608 2925 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 03:26:20.475072 2925 catalog_manager.cc:1504] Loading token signing keys...
I20260810 03:26:20.483520 2925 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b85e7fb7855e4c2282d7b0d2b3a71da9: Generated new TSK 0
I20260810 03:26:20.484534 2925 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 03:26:20.495426 28292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iBH5/build/asan/bin/kudu
/tmp/dist-test-task24iBH5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786332336319544-28292-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786332336319544-28292-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786332336319544-28292-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786332336319544-28292-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.161.1:0
--local_ip_for_outbound_sockets=127.27.161.1
--webserver_interface=127.27.161.1
--webserver_port=0
--tserver_master_addrs=127.27.161.62:42379
--builtin_ntp_servers=127.27.161.20:36911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260810 03:26:20.855075 2944 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:26:20.855417 2944 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:26:20.855540 2944 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:26:20.863667 2944 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:26:20.863796 2944 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:26:20.863950 2944 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.161.1
I20260810 03:26:20.873634 2944 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.161.20:36911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786332336319544-28292-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786332336319544-28292-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.161.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786332336319544-28292-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.161.1
--webserver_port=0
--tserver_master_addrs=127.27.161.62:42379
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.161.1
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786332336319544-28292-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:26:20.875053 2944 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:26:20.876997 2944 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:26:20.890662 2952 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:26:20.890745 2954 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:26:20.890703 2956 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:26:20.891968 2944 server_base.cc:1034] running on GCE node
I20260810 03:26:20.892658 2944 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:26:20.893680 2944 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:26:20.894906 2944 hybrid_clock.cc:648] HybridClock initialized: now 1786332380894848 us; error 49 us; skew 500 ppm
I20260810 03:26:20.895273 2944 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 03:26:20.897805 2944 webserver.cc:460] Webserver started at http://127.27.161.1:37205/ using document root <none> and password file <none>
I20260810 03:26:20.898591 2944 fs_manager.cc:359] Metadata directory not provided
I20260810 03:26:20.898715 2944 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:26:20.899058 2944 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:26:20.902390 2944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786332336319544-28292-0/minicluster-data/ts-0/data/instance:
uuid: "a40e9d3ca975453296f44d142e052a7f"
format_stamp: "Formatted at 2026-08-10 03:26:20 on dist-test-slave-vpvm"
server_key: "74fdb27c38fb00b7c57ded8019b38c01"
server_key_iv: "927ad48b84c6dcf4190b6f63928fa08c"
server_key_version: "encryptionkey@0"
I20260810 03:26:20.903281 2944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786332336319544-28292-0/minicluster-data/ts-0/wal/instance:
uuid: "a40e9d3ca975453296f44d142e052a7f"
format_stamp: "Formatted at 2026-08-10 03:26:20 on dist-test-slave-vpvm"
server_key: "74fdb27c38fb00b7c57ded8019b38c01"
server_key_iv: "927ad48b84c6dcf4190b6f63928fa08c"
server_key_version: "encryptionkey@0"
I20260810 03:26:20.909653 2944 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260810 03:26:20.913766 2966 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:26:20.915388 2944 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260810 03:26:20.915591 2944 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786332336319544-28292-0/minicluster-data/ts-0/data,/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786332336319544-28292-0/minicluster-data/ts-0/wal
uuid: "a40e9d3ca975453296f44d142e052a7f"
format_stamp: "Formatted at 2026-08-10 03:26:20 on dist-test-slave-vpvm"
server_key: "74fdb27c38fb00b7c57ded8019b38c01"
server_key_iv: "927ad48b84c6dcf4190b6f63928fa08c"
server_key_version: "encryptionkey@0"
I20260810 03:26:20.915827 2944 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786332336319544-28292-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786332336319544-28292-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786332336319544-28292-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:26:20.922386 2944 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:26:20.923339 2944 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:26:20.923617 2944 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:26:20.924659 2944 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:26:20.926219 2944 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:26:20.926324 2944 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:26:20.926432 2944 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:26:20.926501 2944 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:26:20.959142 2944 rpc_server.cc:307] RPC server started. Bound to: 127.27.161.1:40381
I20260810 03:26:20.959197 3144 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.161.1:40381 every 8 connection(s)
I20260810 03:26:20.960471 2944 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786332336319544-28292-0/minicluster-data/ts-0/data/info.pb
I20260810 03:26:20.962987 28292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iBH5/build/asan/bin/kudu as pid 2944
I20260810 03:26:20.963232 28292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786332336319544-28292-0/minicluster-data/ts-0/wal/instance
I20260810 03:26:20.964108 28292 external_mini_cluster.cc:1428] Setting key 5ed7985612d12a9def57c7aa3399a62b
I20260810 03:26:20.967535 28292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iBH5/build/asan/bin/kudu
/tmp/dist-test-task24iBH5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786332336319544-28292-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786332336319544-28292-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786332336319544-28292-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786332336319544-28292-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.161.2:0
--local_ip_for_outbound_sockets=127.27.161.2
--webserver_interface=127.27.161.2
--webserver_port=0
--tserver_master_addrs=127.27.161.62:42379
--builtin_ntp_servers=127.27.161.20:36911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260810 03:26:20.971467 3147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:20.963649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.161.1:52229 (local address 127.27.161.62:42379)
0810 03:26:20.964106 (+ 457us) server_negotiation.cc:206] Beginning negotiation
0810 03:26:20.964111 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:26:20.965989 (+ 1878us) server_negotiation.cc:407] Connection header received
0810 03:26:20.968024 (+ 2035us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:26:20.968030 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:26:20.968099 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:26:20.968219 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:26:20.970174 (+ 1955us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:26:20.970183 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:26:20.970187 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:26:20.970221 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:26:20.970300 (+ 79us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:26:20.970319 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:26:20.970506 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0810 03:26:20.970904 (+ 398us) server_negotiation.cc:299] Negotiation successful
0810 03:26:20.971138 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348,"thread_start_us":227,"threads_started":1}
I20260810 03:26:20.972308 3148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:20.964471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:42379 (local address 127.27.161.1:52229)
0810 03:26:20.965771 (+ 1300us) negotiation.cc:107] Waiting for socket to connect
0810 03:26:20.965832 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0810 03:26:20.967653 (+ 1821us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:26:20.968469 (+ 816us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:26:20.968478 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:26:20.968597 (+ 119us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:26:20.969495 (+ 898us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 03:26:20.969499 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:26:20.969575 (+ 76us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 03:26:20.969748 (+ 173us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:26:20.970532 (+ 784us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:26:20.970548 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:26:20.970558 (+ 10us) client_negotiation.cc:769] Sending connection context
0810 03:26:20.970915 (+ 357us) client_negotiation.cc:240] Negotiation successful
0810 03:26:20.971198 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":665,"thread_start_us":410,"threads_started":1}
I20260810 03:26:20.974042 3145 heartbeater.cc:347] Connected to a master server at 127.27.161.62:42379
I20260810 03:26:20.974375 3145 heartbeater.cc:464] Registering TS with master...
I20260810 03:26:20.975039 3145 heartbeater.cc:511] Master 127.27.161.62:42379 requested a full tablet report, sending...
I20260810 03:26:20.977547 2855 ts_manager.cc:194] Registered new tserver with Master: a40e9d3ca975453296f44d142e052a7f (127.27.161.1:40381)
W20260810 03:26:21.318739 3149 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:26:21.319067 3149 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:26:21.319197 3149 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:26:21.327054 3149 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:26:21.327180 3149 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:26:21.327324 3149 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.161.2
I20260810 03:26:21.336934 3149 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.161.20:36911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786332336319544-28292-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786332336319544-28292-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.161.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786332336319544-28292-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.161.2
--webserver_port=0
--tserver_master_addrs=127.27.161.62:42379
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.161.2
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786332336319544-28292-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:26:21.338379 3149 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:26:21.340257 3149 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:26:21.353595 3163 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:26:21.353571 3161 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:26:21.353639 3159 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:26:21.354423 3149 server_base.cc:1034] running on GCE node
I20260810 03:26:21.355033 3149 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:26:21.356096 3149 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:26:21.357290 3149 hybrid_clock.cc:648] HybridClock initialized: now 1786332381357214 us; error 68 us; skew 500 ppm
I20260810 03:26:21.357614 3149 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 03:26:21.359885 3149 webserver.cc:460] Webserver started at http://127.27.161.2:36689/ using document root <none> and password file <none>
I20260810 03:26:21.360601 3149 fs_manager.cc:359] Metadata directory not provided
I20260810 03:26:21.360702 3149 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:26:21.361029 3149 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:26:21.364285 3149 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786332336319544-28292-0/minicluster-data/ts-1/data/instance:
uuid: "43dcd84a2dd742bbb7ee640b5638022b"
format_stamp: "Formatted at 2026-08-10 03:26:21 on dist-test-slave-vpvm"
server_key: "2c05352e6f8b7d7875ff9439ab028876"
server_key_iv: "70fcef17092717d8cf4373df4c0fd346"
server_key_version: "encryptionkey@0"
I20260810 03:26:21.365128 3149 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786332336319544-28292-0/minicluster-data/ts-1/wal/instance:
uuid: "43dcd84a2dd742bbb7ee640b5638022b"
format_stamp: "Formatted at 2026-08-10 03:26:21 on dist-test-slave-vpvm"
server_key: "2c05352e6f8b7d7875ff9439ab028876"
server_key_iv: "70fcef17092717d8cf4373df4c0fd346"
server_key_version: "encryptionkey@0"
I20260810 03:26:21.370949 3149 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260810 03:26:21.374717 3169 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:26:21.376212 3149 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 03:26:21.376390 3149 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786332336319544-28292-0/minicluster-data/ts-1/data,/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786332336319544-28292-0/minicluster-data/ts-1/wal
uuid: "43dcd84a2dd742bbb7ee640b5638022b"
format_stamp: "Formatted at 2026-08-10 03:26:21 on dist-test-slave-vpvm"
server_key: "2c05352e6f8b7d7875ff9439ab028876"
server_key_iv: "70fcef17092717d8cf4373df4c0fd346"
server_key_version: "encryptionkey@0"
I20260810 03:26:21.376617 3149 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786332336319544-28292-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786332336319544-28292-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786332336319544-28292-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:26:21.382603 3149 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:26:21.383496 3149 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:26:21.383769 3149 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:26:21.384855 3149 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:26:21.386440 3149 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:26:21.386541 3149 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:26:21.386660 3149 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:26:21.386740 3149 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:26:21.419521 3149 rpc_server.cc:307] RPC server started. Bound to: 127.27.161.2:41367
I20260810 03:26:21.419566 3338 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.161.2:41367 every 8 connection(s)
I20260810 03:26:21.420878 3149 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786332336319544-28292-0/minicluster-data/ts-1/data/info.pb
I20260810 03:26:21.426998 28292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iBH5/build/asan/bin/kudu as pid 3149
I20260810 03:26:21.427186 28292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786332336319544-28292-0/minicluster-data/ts-1/wal/instance
I20260810 03:26:21.427765 28292 external_mini_cluster.cc:1428] Setting key 062f1f0445a157525fd5be138128a25c
I20260810 03:26:21.430846 3147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:21.423825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.161.2:51799 (local address 127.27.161.62:42379)
0810 03:26:21.424056 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0810 03:26:21.424060 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:26:21.425710 (+ 1650us) server_negotiation.cc:407] Connection header received
0810 03:26:21.428120 (+ 2410us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:26:21.428127 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:26:21.428220 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:26:21.428323 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:26:21.430077 (+ 1754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:26:21.430083 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:26:21.430088 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:26:21.430108 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:26:21.430146 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:26:21.430158 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:26:21.430271 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0810 03:26:21.430535 (+ 264us) server_negotiation.cc:299] Negotiation successful
0810 03:26:21.430590 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260810 03:26:21.431368 28292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iBH5/build/asan/bin/kudu
/tmp/dist-test-task24iBH5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786332336319544-28292-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786332336319544-28292-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786332336319544-28292-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786332336319544-28292-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.161.3:0
--local_ip_for_outbound_sockets=127.27.161.3
--webserver_interface=127.27.161.3
--webserver_port=0
--tserver_master_addrs=127.27.161.62:42379
--builtin_ntp_servers=127.27.161.20:36911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260810 03:26:21.432300 3342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:21.424372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:42379 (local address 127.27.161.2:51799)
0810 03:26:21.425496 (+ 1124us) negotiation.cc:107] Waiting for socket to connect
0810 03:26:21.425544 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0810 03:26:21.427800 (+ 2256us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:26:21.428811 (+ 1011us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:26:21.428825 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:26:21.428964 (+ 139us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:26:21.429685 (+ 721us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 03:26:21.429693 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:26:21.429753 (+ 60us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 03:26:21.429886 (+ 133us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:26:21.430288 (+ 402us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:26:21.430302 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:26:21.430311 (+ 9us) client_negotiation.cc:769] Sending connection context
0810 03:26:21.430634 (+ 323us) client_negotiation.cc:240] Negotiation successful
0810 03:26:21.430948 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":545,"thread_start_us":329,"threads_started":1}
I20260810 03:26:21.433835 3339 heartbeater.cc:347] Connected to a master server at 127.27.161.62:42379
I20260810 03:26:21.434202 3339 heartbeater.cc:464] Registering TS with master...
I20260810 03:26:21.434859 3339 heartbeater.cc:511] Master 127.27.161.62:42379 requested a full tablet report, sending...
I20260810 03:26:21.436419 2855 ts_manager.cc:194] Registered new tserver with Master: 43dcd84a2dd742bbb7ee640b5638022b (127.27.161.2:41367)
W20260810 03:26:21.789140 3344 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:26:21.789453 3344 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:26:21.789549 3344 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:26:21.797015 3344 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:26:21.797155 3344 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:26:21.797297 3344 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.161.3
I20260810 03:26:21.806907 3344 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.161.20:36911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786332336319544-28292-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786332336319544-28292-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.161.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786332336319544-28292-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.161.3
--webserver_port=0
--tserver_master_addrs=127.27.161.62:42379
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.161.3
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786332336319544-28292-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:26:21.808318 3344 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:26:21.810250 3344 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:26:21.823994 3349 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:26:21.824021 3350 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:26:21.824156 3355 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:26:21.825049 3344 server_base.cc:1034] running on GCE node
I20260810 03:26:21.825704 3344 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:26:21.826786 3344 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:26:21.828020 3344 hybrid_clock.cc:648] HybridClock initialized: now 1786332381827956 us; error 49 us; skew 500 ppm
I20260810 03:26:21.828361 3344 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 03:26:21.830876 3344 webserver.cc:460] Webserver started at http://127.27.161.3:38285/ using document root <none> and password file <none>
I20260810 03:26:21.831643 3344 fs_manager.cc:359] Metadata directory not provided
I20260810 03:26:21.831753 3344 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:26:21.832103 3344 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:26:21.835217 3344 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786332336319544-28292-0/minicluster-data/ts-2/data/instance:
uuid: "0d04a6257eb74f7f8ad11e16070dcf81"
format_stamp: "Formatted at 2026-08-10 03:26:21 on dist-test-slave-vpvm"
server_key: "b5aa9efd14b587ad18f55725cd5ba041"
server_key_iv: "9229f8917ed579130e0da81cde16ec45"
server_key_version: "encryptionkey@0"
I20260810 03:26:21.836112 3344 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786332336319544-28292-0/minicluster-data/ts-2/wal/instance:
uuid: "0d04a6257eb74f7f8ad11e16070dcf81"
format_stamp: "Formatted at 2026-08-10 03:26:21 on dist-test-slave-vpvm"
server_key: "b5aa9efd14b587ad18f55725cd5ba041"
server_key_iv: "9229f8917ed579130e0da81cde16ec45"
server_key_version: "encryptionkey@0"
I20260810 03:26:21.841862 3344 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.003s
I20260810 03:26:21.845654 3364 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:26:21.847142 3344 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260810 03:26:21.847317 3344 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786332336319544-28292-0/minicluster-data/ts-2/data,/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786332336319544-28292-0/minicluster-data/ts-2/wal
uuid: "0d04a6257eb74f7f8ad11e16070dcf81"
format_stamp: "Formatted at 2026-08-10 03:26:21 on dist-test-slave-vpvm"
server_key: "b5aa9efd14b587ad18f55725cd5ba041"
server_key_iv: "9229f8917ed579130e0da81cde16ec45"
server_key_version: "encryptionkey@0"
I20260810 03:26:21.847540 3344 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786332336319544-28292-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786332336319544-28292-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786332336319544-28292-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:26:21.853646 3344 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:26:21.854574 3344 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:26:21.854836 3344 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:26:21.855928 3344 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:26:21.857484 3344 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:26:21.857575 3344 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:26:21.857679 3344 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:26:21.857831 3344 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:26:21.891760 3344 rpc_server.cc:307] RPC server started. Bound to: 127.27.161.3:35323
I20260810 03:26:21.891795 3545 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.161.3:35323 every 8 connection(s)
I20260810 03:26:21.893133 3344 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786332336319544-28292-0/minicluster-data/ts-2/data/info.pb
I20260810 03:26:21.899441 28292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iBH5/build/asan/bin/kudu as pid 3344
I20260810 03:26:21.899644 28292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786332336319544-28292-0/minicluster-data/ts-2/wal/instance
I20260810 03:26:21.900096 28292 external_mini_cluster.cc:1428] Setting key 9f80b4d73e9fad8732df7d0fe7718a6b
I20260810 03:26:21.902021 3147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:21.895905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.161.3:56865 (local address 127.27.161.62:42379)
0810 03:26:21.896140 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0810 03:26:21.896147 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:26:21.897764 (+ 1617us) server_negotiation.cc:407] Connection header received
0810 03:26:21.899582 (+ 1818us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:26:21.899588 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:26:21.899663 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:26:21.899756 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:26:21.901211 (+ 1455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:26:21.901216 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:26:21.901219 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:26:21.901237 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:26:21.901271 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:26:21.901282 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:26:21.901396 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0810 03:26:21.901670 (+ 274us) server_negotiation.cc:299] Negotiation successful
0810 03:26:21.901732 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260810 03:26:21.903182 3552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:21.896486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:42379 (local address 127.27.161.3:56865)
0810 03:26:21.897570 (+ 1084us) negotiation.cc:107] Waiting for socket to connect
0810 03:26:21.897625 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0810 03:26:21.899277 (+ 1652us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:26:21.900046 (+ 769us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:26:21.900055 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:26:21.900154 (+ 99us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:26:21.900887 (+ 733us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 03:26:21.900890 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:26:21.900935 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 03:26:21.901037 (+ 102us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:26:21.901396 (+ 359us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:26:21.901407 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:26:21.901415 (+ 8us) client_negotiation.cc:769] Sending connection context
0810 03:26:21.901746 (+ 331us) client_negotiation.cc:240] Negotiation successful
0810 03:26:21.902078 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":556,"thread_start_us":336,"threads_started":1}
I20260810 03:26:21.904623 3548 heartbeater.cc:347] Connected to a master server at 127.27.161.62:42379
I20260810 03:26:21.904960 3548 heartbeater.cc:464] Registering TS with master...
I20260810 03:26:21.905632 3548 heartbeater.cc:511] Master 127.27.161.62:42379 requested a full tablet report, sending...
I20260810 03:26:21.907233 2855 ts_manager.cc:194] Registered new tserver with Master: 0d04a6257eb74f7f8ad11e16070dcf81 (127.27.161.3:35323)
I20260810 03:26:21.916539 28292 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 03:26:21.924343 3147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:21.921472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43362 (local address 127.27.161.62:42379)
0810 03:26:21.921721 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0810 03:26:21.921728 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:26:21.922263 (+ 535us) server_negotiation.cc:407] Connection header received
0810 03:26:21.922457 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:26:21.922461 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:26:21.922528 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:26:21.922617 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:26:21.923661 (+ 1044us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:26:21.923665 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:26:21.923669 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:26:21.923685 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:26:21.923719 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:26:21.923731 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:26:21.923864 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0810 03:26:21.923954 (+ 90us) server_negotiation.cc:299] Negotiation successful
0810 03:26:21.924032 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
I20260810 03:26:21.928939 2855 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43362:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 03:26:21.936023 2855 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 03:26:21.969913 3563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:21.949287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.2:41367 (local address 127.0.0.1:44790)
0810 03:26:21.955900 (+ 6613us) negotiation.cc:107] Waiting for socket to connect
0810 03:26:21.955948 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0810 03:26:21.956106 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:26:21.966240 (+ 10134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:26:21.966250 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:26:21.966388 (+ 138us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:26:21.967393 (+ 1005us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 03:26:21.967398 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:26:21.967436 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 03:26:21.967552 (+ 116us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:26:21.969034 (+ 1482us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:26:21.969049 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:26:21.969061 (+ 12us) client_negotiation.cc:769] Sending connection context
0810 03:26:21.969269 (+ 208us) client_negotiation.cc:240] Negotiation successful
0810 03:26:21.969386 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5910,"thread_start_us":5708,"threads_started":1}
I20260810 03:26:21.971283 3564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:21.949634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44790 (local address 127.27.161.2:41367)
0810 03:26:21.965212 (+ 15578us) server_negotiation.cc:206] Beginning negotiation
0810 03:26:21.965233 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0810 03:26:21.965275 (+ 42us) server_negotiation.cc:407] Connection header received
0810 03:26:21.965360 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:26:21.965368 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:26:21.965638 (+ 270us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:26:21.967005 (+ 1367us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:26:21.968733 (+ 1728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:26:21.968744 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:26:21.968761 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:26:21.968788 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:26:21.968847 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:26:21.968868 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:26:21.970674 (+ 1806us) server_negotiation.cc:1035] Waiting for connection context
0810 03:26:21.970769 (+ 95us) server_negotiation.cc:299] Negotiation successful
0810 03:26:21.970897 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15364,"thread_start_us":16192,"threads_started":1}
I20260810 03:26:21.975241 3563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:21.949497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.1:40381 (local address 127.0.0.1:46910)
0810 03:26:21.970250 (+ 20753us) negotiation.cc:107] Waiting for socket to connect
0810 03:26:21.970273 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0810 03:26:21.970381 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:26:21.972788 (+ 2407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:26:21.972794 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:26:21.972842 (+ 48us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:26:21.973454 (+ 612us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 03:26:21.973458 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:26:21.973488 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 03:26:21.973542 (+ 54us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:26:21.974752 (+ 1210us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:26:21.974762 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:26:21.974767 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 03:26:21.974860 (+ 93us) client_negotiation.cc:240] Negotiation successful
0810 03:26:21.974927 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20624,"thread_start_us":25485,"threads_started":1}
I20260810 03:26:21.976262 3567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:21.950157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46910 (local address 127.27.161.1:40381)
0810 03:26:21.971937 (+ 21780us) server_negotiation.cc:206] Beginning negotiation
0810 03:26:21.971947 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 03:26:21.971980 (+ 33us) server_negotiation.cc:407] Connection header received
0810 03:26:21.972064 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:26:21.972074 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:26:21.972334 (+ 260us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:26:21.973166 (+ 832us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:26:21.974466 (+ 1300us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:26:21.974474 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:26:21.974500 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:26:21.974519 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:26:21.974589 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:26:21.974613 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:26:21.975695 (+ 1082us) server_negotiation.cc:1035] Waiting for connection context
0810 03:26:21.975782 (+ 87us) server_negotiation.cc:299] Negotiation successful
0810 03:26:21.975892 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21589,"thread_start_us":22310,"threads_started":1}
I20260810 03:26:21.976507 3572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:21.960527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.3:35323 (local address 127.0.0.1:54964)
0810 03:26:21.972102 (+ 11575us) negotiation.cc:107] Waiting for socket to connect
0810 03:26:21.972120 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 03:26:21.972222 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:26:21.974482 (+ 2260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:26:21.974488 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:26:21.974513 (+ 25us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:26:21.975292 (+ 779us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 03:26:21.975296 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:26:21.975313 (+ 17us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 03:26:21.975358 (+ 45us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:26:21.975909 (+ 551us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:26:21.975914 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:26:21.975919 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 03:26:21.976065 (+ 146us) client_negotiation.cc:240] Negotiation successful
0810 03:26:21.976119 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11419,"thread_start_us":11272,"threads_started":1}
I20260810 03:26:21.977366 3569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:21.956314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54964 (local address 127.27.161.3:35323)
0810 03:26:21.972794 (+ 16480us) server_negotiation.cc:206] Beginning negotiation
0810 03:26:21.972804 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 03:26:21.972839 (+ 35us) server_negotiation.cc:407] Connection header received
0810 03:26:21.972914 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:26:21.972922 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:26:21.973142 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:26:21.973299 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:26:21.975552 (+ 2253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:26:21.975561 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:26:21.975575 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:26:21.975596 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:26:21.975661 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:26:21.975685 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:26:21.975848 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0810 03:26:21.976881 (+ 1033us) server_negotiation.cc:299] Negotiation successful
0810 03:26:21.976994 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16287,"thread_start_us":20153,"threads_started":1}
I20260810 03:26:21.977228 3235 tablet_service.cc:1467] Processing CreateTablet for tablet 6c4417932190447ea1a9364f4014ba28 (DEFAULT_TABLE table=test-table [id=b96259704a4a4b8082c85dbf605835fd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:26:21.978919 3235 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c4417932190447ea1a9364f4014ba28. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:26:21.980798 3145 heartbeater.cc:503] Master 127.27.161.62:42379 was elected leader, sending a full tablet report...
I20260810 03:26:21.980505 3041 tablet_service.cc:1467] Processing CreateTablet for tablet 6c4417932190447ea1a9364f4014ba28 (DEFAULT_TABLE table=test-table [id=b96259704a4a4b8082c85dbf605835fd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:26:21.980804 3445 tablet_service.cc:1467] Processing CreateTablet for tablet 6c4417932190447ea1a9364f4014ba28 (DEFAULT_TABLE table=test-table [id=b96259704a4a4b8082c85dbf605835fd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:26:21.982069 3041 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c4417932190447ea1a9364f4014ba28. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:26:21.982249 3445 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c4417932190447ea1a9364f4014ba28. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:26:21.994954 3573 tablet_bootstrap.cc:492] T 6c4417932190447ea1a9364f4014ba28 P 43dcd84a2dd742bbb7ee640b5638022b: Bootstrap starting.
I20260810 03:26:21.997229 3574 tablet_bootstrap.cc:492] T 6c4417932190447ea1a9364f4014ba28 P 0d04a6257eb74f7f8ad11e16070dcf81: Bootstrap starting.
I20260810 03:26:21.999704 3573 tablet_bootstrap.cc:654] T 6c4417932190447ea1a9364f4014ba28 P 43dcd84a2dd742bbb7ee640b5638022b: Neither blocks nor log segments found. Creating new log.
I20260810 03:26:21.999889 3575 tablet_bootstrap.cc:492] T 6c4417932190447ea1a9364f4014ba28 P a40e9d3ca975453296f44d142e052a7f: Bootstrap starting.
I20260810 03:26:22.001192 3573 log.cc:824] T 6c4417932190447ea1a9364f4014ba28 P 43dcd84a2dd742bbb7ee640b5638022b: Log is configured to *not* fsync() on all Append() calls
I20260810 03:26:22.001543 3574 tablet_bootstrap.cc:654] T 6c4417932190447ea1a9364f4014ba28 P 0d04a6257eb74f7f8ad11e16070dcf81: Neither blocks nor log segments found. Creating new log.
I20260810 03:26:22.002843 3574 log.cc:824] T 6c4417932190447ea1a9364f4014ba28 P 0d04a6257eb74f7f8ad11e16070dcf81: Log is configured to *not* fsync() on all Append() calls
I20260810 03:26:22.003630 3575 tablet_bootstrap.cc:654] T 6c4417932190447ea1a9364f4014ba28 P a40e9d3ca975453296f44d142e052a7f: Neither blocks nor log segments found. Creating new log.
I20260810 03:26:22.004879 3573 tablet_bootstrap.cc:492] T 6c4417932190447ea1a9364f4014ba28 P 43dcd84a2dd742bbb7ee640b5638022b: No bootstrap required, opened a new log
I20260810 03:26:22.004952 3575 log.cc:824] T 6c4417932190447ea1a9364f4014ba28 P a40e9d3ca975453296f44d142e052a7f: Log is configured to *not* fsync() on all Append() calls
I20260810 03:26:22.005225 3573 ts_tablet_manager.cc:1397] T 6c4417932190447ea1a9364f4014ba28 P 43dcd84a2dd742bbb7ee640b5638022b: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20260810 03:26:22.005524 3574 tablet_bootstrap.cc:492] T 6c4417932190447ea1a9364f4014ba28 P 0d04a6257eb74f7f8ad11e16070dcf81: No bootstrap required, opened a new log
I20260810 03:26:22.005801 3574 ts_tablet_manager.cc:1397] T 6c4417932190447ea1a9364f4014ba28 P 0d04a6257eb74f7f8ad11e16070dcf81: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20260810 03:26:22.013306 3575 tablet_bootstrap.cc:492] T 6c4417932190447ea1a9364f4014ba28 P a40e9d3ca975453296f44d142e052a7f: No bootstrap required, opened a new log
I20260810 03:26:22.013746 3575 ts_tablet_manager.cc:1397] T 6c4417932190447ea1a9364f4014ba28 P a40e9d3ca975453296f44d142e052a7f: Time spent bootstrapping tablet: real 0.014s user 0.013s sys 0.000s
I20260810 03:26:22.013576 3574 raft_consensus.cc:348] T 6c4417932190447ea1a9364f4014ba28 P 0d04a6257eb74f7f8ad11e16070dcf81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43dcd84a2dd742bbb7ee640b5638022b" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 41367 } } peers { permanent_uuid: "a40e9d3ca975453296f44d142e052a7f" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 40381 } } peers { permanent_uuid: "0d04a6257eb74f7f8ad11e16070dcf81" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 35323 } }
I20260810 03:26:22.014041 3574 raft_consensus.cc:374] T 6c4417932190447ea1a9364f4014ba28 P 0d04a6257eb74f7f8ad11e16070dcf81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:26:22.014151 3574 raft_consensus.cc:729] T 6c4417932190447ea1a9364f4014ba28 P 0d04a6257eb74f7f8ad11e16070dcf81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d04a6257eb74f7f8ad11e16070dcf81, State: Initialized, Role: FOLLOWER
I20260810 03:26:22.014948 3574 consensus_queue.cc:260] T 6c4417932190447ea1a9364f4014ba28 P 0d04a6257eb74f7f8ad11e16070dcf81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43dcd84a2dd742bbb7ee640b5638022b" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 41367 } } peers { permanent_uuid: "a40e9d3ca975453296f44d142e052a7f" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 40381 } } peers { permanent_uuid: "0d04a6257eb74f7f8ad11e16070dcf81" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 35323 } }
I20260810 03:26:22.015782 3573 raft_consensus.cc:348] T 6c4417932190447ea1a9364f4014ba28 P 43dcd84a2dd742bbb7ee640b5638022b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43dcd84a2dd742bbb7ee640b5638022b" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 41367 } } peers { permanent_uuid: "a40e9d3ca975453296f44d142e052a7f" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 40381 } } peers { permanent_uuid: "0d04a6257eb74f7f8ad11e16070dcf81" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 35323 } }
I20260810 03:26:22.016323 3573 raft_consensus.cc:374] T 6c4417932190447ea1a9364f4014ba28 P 43dcd84a2dd742bbb7ee640b5638022b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:26:22.016461 3573 raft_consensus.cc:729] T 6c4417932190447ea1a9364f4014ba28 P 43dcd84a2dd742bbb7ee640b5638022b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43dcd84a2dd742bbb7ee640b5638022b, State: Initialized, Role: FOLLOWER
I20260810 03:26:22.016834 3548 heartbeater.cc:503] Master 127.27.161.62:42379 was elected leader, sending a full tablet report...
I20260810 03:26:22.017243 3573 consensus_queue.cc:260] T 6c4417932190447ea1a9364f4014ba28 P 43dcd84a2dd742bbb7ee640b5638022b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43dcd84a2dd742bbb7ee640b5638022b" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 41367 } } peers { permanent_uuid: "a40e9d3ca975453296f44d142e052a7f" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 40381 } } peers { permanent_uuid: "0d04a6257eb74f7f8ad11e16070dcf81" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 35323 } }
I20260810 03:26:22.017571 3574 ts_tablet_manager.cc:1428] T 6c4417932190447ea1a9364f4014ba28 P 0d04a6257eb74f7f8ad11e16070dcf81: Time spent starting tablet: real 0.012s user 0.003s sys 0.007s
I20260810 03:26:22.019614 3339 heartbeater.cc:503] Master 127.27.161.62:42379 was elected leader, sending a full tablet report...
I20260810 03:26:22.020390 3573 ts_tablet_manager.cc:1428] T 6c4417932190447ea1a9364f4014ba28 P 43dcd84a2dd742bbb7ee640b5638022b: Time spent starting tablet: real 0.015s user 0.000s sys 0.017s
I20260810 03:26:22.025321 3575 raft_consensus.cc:348] T 6c4417932190447ea1a9364f4014ba28 P a40e9d3ca975453296f44d142e052a7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43dcd84a2dd742bbb7ee640b5638022b" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 41367 } } peers { permanent_uuid: "a40e9d3ca975453296f44d142e052a7f" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 40381 } } peers { permanent_uuid: "0d04a6257eb74f7f8ad11e16070dcf81" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 35323 } }
I20260810 03:26:22.025722 3575 raft_consensus.cc:374] T 6c4417932190447ea1a9364f4014ba28 P a40e9d3ca975453296f44d142e052a7f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:26:22.025888 3575 raft_consensus.cc:729] T 6c4417932190447ea1a9364f4014ba28 P a40e9d3ca975453296f44d142e052a7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a40e9d3ca975453296f44d142e052a7f, State: Initialized, Role: FOLLOWER
I20260810 03:26:22.026656 3575 consensus_queue.cc:260] T 6c4417932190447ea1a9364f4014ba28 P a40e9d3ca975453296f44d142e052a7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43dcd84a2dd742bbb7ee640b5638022b" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 41367 } } peers { permanent_uuid: "a40e9d3ca975453296f44d142e052a7f" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 40381 } } peers { permanent_uuid: "0d04a6257eb74f7f8ad11e16070dcf81" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 35323 } }
I20260810 03:26:22.028556 3575 ts_tablet_manager.cc:1428] T 6c4417932190447ea1a9364f4014ba28 P a40e9d3ca975453296f44d142e052a7f: Time spent starting tablet: real 0.015s user 0.008s sys 0.004s
I20260810 03:26:22.048473 3586 raft_consensus.cc:482] T 6c4417932190447ea1a9364f4014ba28 P a40e9d3ca975453296f44d142e052a7f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 03:26:22.048820 3586 raft_consensus.cc:504] T 6c4417932190447ea1a9364f4014ba28 P a40e9d3ca975453296f44d142e052a7f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43dcd84a2dd742bbb7ee640b5638022b" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 41367 } } peers { permanent_uuid: "a40e9d3ca975453296f44d142e052a7f" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 40381 } } peers { permanent_uuid: "0d04a6257eb74f7f8ad11e16070dcf81" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 35323 } }
I20260810 03:26:22.050678 3586 leader_election.cc:288] T 6c4417932190447ea1a9364f4014ba28 P a40e9d3ca975453296f44d142e052a7f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 43dcd84a2dd742bbb7ee640b5638022b (127.27.161.2:41367), 0d04a6257eb74f7f8ad11e16070dcf81 (127.27.161.3:35323)
I20260810 03:26:22.053972 3564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:22.051230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.161.1:58475 (local address 127.27.161.2:41367)
0810 03:26:22.051469 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0810 03:26:22.051475 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:26:22.051681 (+ 206us) server_negotiation.cc:407] Connection header received
0810 03:26:22.051905 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:26:22.051912 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:26:22.051984 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:26:22.052065 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:26:22.053300 (+ 1235us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:26:22.053305 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:26:22.053308 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:26:22.053325 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:26:22.053356 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:26:22.053368 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:26:22.053471 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0810 03:26:22.053581 (+ 110us) server_negotiation.cc:299] Negotiation successful
0810 03:26:22.053653 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260810 03:26:22.054155 3587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:22.050927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.2:41367 (local address 127.27.161.1:58475)
0810 03:26:22.051552 (+ 625us) negotiation.cc:107] Waiting for socket to connect
0810 03:26:22.051573 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0810 03:26:22.051730 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:26:22.052295 (+ 565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:26:22.052299 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:26:22.052328 (+ 29us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:26:22.053073 (+ 745us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 03:26:22.053076 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:26:22.053095 (+ 19us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 03:26:22.053147 (+ 52us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:26:22.053463 (+ 316us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:26:22.053468 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:26:22.053473 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 03:26:22.053625 (+ 152us) client_negotiation.cc:240] Negotiation successful
0810 03:26:22.053680 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":310,"threads_started":1}
I20260810 03:26:22.054455 3569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:22.051969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.161.1:39727 (local address 127.27.161.3:35323)
0810 03:26:22.052185 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0810 03:26:22.052189 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:26:22.052437 (+ 248us) server_negotiation.cc:407] Connection header received
0810 03:26:22.052571 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:26:22.052576 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:26:22.052653 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:26:22.052756 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:26:22.053740 (+ 984us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:26:22.053745 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:26:22.053749 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:26:22.053768 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:26:22.053803 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:26:22.053816 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:26:22.053944 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0810 03:26:22.054100 (+ 156us) server_negotiation.cc:299] Negotiation successful
0810 03:26:22.054172 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
I20260810 03:26:22.054775 3588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:22.051680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.3:35323 (local address 127.27.161.1:39727)
0810 03:26:22.052318 (+ 638us) negotiation.cc:107] Waiting for socket to connect
0810 03:26:22.052336 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 03:26:22.052450 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:26:22.052779 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:26:22.052783 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:26:22.052803 (+ 20us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:26:22.053505 (+ 702us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 03:26:22.053508 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:26:22.053523 (+ 15us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 03:26:22.053567 (+ 44us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:26:22.053920 (+ 353us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:26:22.053926 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:26:22.053936 (+ 10us) client_negotiation.cc:769] Sending connection context
0810 03:26:22.054152 (+ 216us) client_negotiation.cc:240] Negotiation successful
0810 03:26:22.054206 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":513,"thread_start_us":290,"threads_started":1}
I20260810 03:26:22.055392 3273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c4417932190447ea1a9364f4014ba28" candidate_uuid: "a40e9d3ca975453296f44d142e052a7f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43dcd84a2dd742bbb7ee640b5638022b" is_pre_election: true
I20260810 03:26:22.056094 3273 raft_consensus.cc:2454] T 6c4417932190447ea1a9364f4014ba28 P 43dcd84a2dd742bbb7ee640b5638022b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a40e9d3ca975453296f44d142e052a7f in term 0.
I20260810 03:26:22.056010 3474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c4417932190447ea1a9364f4014ba28" candidate_uuid: "a40e9d3ca975453296f44d142e052a7f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d04a6257eb74f7f8ad11e16070dcf81" is_pre_election: true
I20260810 03:26:22.056591 3474 raft_consensus.cc:2454] T 6c4417932190447ea1a9364f4014ba28 P 0d04a6257eb74f7f8ad11e16070dcf81 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a40e9d3ca975453296f44d142e052a7f in term 0.
I20260810 03:26:22.056931 2972 leader_election.cc:302] T 6c4417932190447ea1a9364f4014ba28 P a40e9d3ca975453296f44d142e052a7f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 43dcd84a2dd742bbb7ee640b5638022b, a40e9d3ca975453296f44d142e052a7f; no voters:
I20260810 03:26:22.057411 3586 raft_consensus.cc:2781] T 6c4417932190447ea1a9364f4014ba28 P a40e9d3ca975453296f44d142e052a7f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 03:26:22.057596 3586 raft_consensus.cc:482] T 6c4417932190447ea1a9364f4014ba28 P a40e9d3ca975453296f44d142e052a7f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 03:26:22.057732 3586 raft_consensus.cc:3037] T 6c4417932190447ea1a9364f4014ba28 P a40e9d3ca975453296f44d142e052a7f [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:26:22.060221 3586 raft_consensus.cc:504] T 6c4417932190447ea1a9364f4014ba28 P a40e9d3ca975453296f44d142e052a7f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43dcd84a2dd742bbb7ee640b5638022b" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 41367 } } peers { permanent_uuid: "a40e9d3ca975453296f44d142e052a7f" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 40381 } } peers { permanent_uuid: "0d04a6257eb74f7f8ad11e16070dcf81" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 35323 } }
I20260810 03:26:22.060998 3586 leader_election.cc:288] T 6c4417932190447ea1a9364f4014ba28 P a40e9d3ca975453296f44d142e052a7f [CANDIDATE]: Term 1 election: Requested vote from peers 43dcd84a2dd742bbb7ee640b5638022b (127.27.161.2:41367), 0d04a6257eb74f7f8ad11e16070dcf81 (127.27.161.3:35323)
I20260810 03:26:22.061614 3273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c4417932190447ea1a9364f4014ba28" candidate_uuid: "a40e9d3ca975453296f44d142e052a7f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43dcd84a2dd742bbb7ee640b5638022b"
I20260810 03:26:22.061656 3474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c4417932190447ea1a9364f4014ba28" candidate_uuid: "a40e9d3ca975453296f44d142e052a7f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d04a6257eb74f7f8ad11e16070dcf81"
I20260810 03:26:22.061879 3474 raft_consensus.cc:3037] T 6c4417932190447ea1a9364f4014ba28 P 0d04a6257eb74f7f8ad11e16070dcf81 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:26:22.061879 3273 raft_consensus.cc:3037] T 6c4417932190447ea1a9364f4014ba28 P 43dcd84a2dd742bbb7ee640b5638022b [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:26:22.065284 3273 raft_consensus.cc:2454] T 6c4417932190447ea1a9364f4014ba28 P 43dcd84a2dd742bbb7ee640b5638022b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a40e9d3ca975453296f44d142e052a7f in term 1.
I20260810 03:26:22.065284 3474 raft_consensus.cc:2454] T 6c4417932190447ea1a9364f4014ba28 P 0d04a6257eb74f7f8ad11e16070dcf81 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a40e9d3ca975453296f44d142e052a7f in term 1.
I20260810 03:26:22.065795 2972 leader_election.cc:302] T 6c4417932190447ea1a9364f4014ba28 P a40e9d3ca975453296f44d142e052a7f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d04a6257eb74f7f8ad11e16070dcf81, a40e9d3ca975453296f44d142e052a7f; no voters:
I20260810 03:26:22.066124 3586 raft_consensus.cc:2781] T 6c4417932190447ea1a9364f4014ba28 P a40e9d3ca975453296f44d142e052a7f [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:26:22.066649 3586 raft_consensus.cc:686] T 6c4417932190447ea1a9364f4014ba28 P a40e9d3ca975453296f44d142e052a7f [term 1 LEADER]: Becoming Leader. State: Replica: a40e9d3ca975453296f44d142e052a7f, State: Running, Role: LEADER
I20260810 03:26:22.067153 3586 consensus_queue.cc:237] T 6c4417932190447ea1a9364f4014ba28 P a40e9d3ca975453296f44d142e052a7f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43dcd84a2dd742bbb7ee640b5638022b" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 41367 } } peers { permanent_uuid: "a40e9d3ca975453296f44d142e052a7f" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 40381 } } peers { permanent_uuid: "0d04a6257eb74f7f8ad11e16070dcf81" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 35323 } }
I20260810 03:26:22.072407 2854 catalog_manager.cc:5495] T 6c4417932190447ea1a9364f4014ba28 P a40e9d3ca975453296f44d142e052a7f reported cstate change: term changed from 0 to 1, leader changed from <none> to a40e9d3ca975453296f44d142e052a7f (127.27.161.1). New cstate: current_term: 1 leader_uuid: "a40e9d3ca975453296f44d142e052a7f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43dcd84a2dd742bbb7ee640b5638022b" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 41367 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a40e9d3ca975453296f44d142e052a7f" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 40381 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0d04a6257eb74f7f8ad11e16070dcf81" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 35323 } health_report { overall_health: UNKNOWN } } }
I20260810 03:26:22.109616 3567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:22.107640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46922 (local address 127.27.161.1:40381)
0810 03:26:22.107851 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0810 03:26:22.107857 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:26:22.107891 (+ 34us) server_negotiation.cc:407] Connection header received
0810 03:26:22.108010 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:26:22.108015 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:26:22.108099 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:26:22.108199 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:26:22.109031 (+ 832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:26:22.109036 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:26:22.109039 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:26:22.109054 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:26:22.109087 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:26:22.109099 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:26:22.109207 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0810 03:26:22.109307 (+ 100us) server_negotiation.cc:299] Negotiation successful
0810 03:26:22.109367 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260810 03:26:22.118296 3474 raft_consensus.cc:1261] T 6c4417932190447ea1a9364f4014ba28 P 0d04a6257eb74f7f8ad11e16070dcf81 [term 1 FOLLOWER]: Refusing update from remote peer a40e9d3ca975453296f44d142e052a7f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:26:22.118296 3273 raft_consensus.cc:1261] T 6c4417932190447ea1a9364f4014ba28 P 43dcd84a2dd742bbb7ee640b5638022b [term 1 FOLLOWER]: Refusing update from remote peer a40e9d3ca975453296f44d142e052a7f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:26:22.119344 3589 consensus_queue.cc:1035] T 6c4417932190447ea1a9364f4014ba28 P a40e9d3ca975453296f44d142e052a7f [LEADER]: Connected to new peer: Peer: permanent_uuid: "43dcd84a2dd742bbb7ee640b5638022b" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 41367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:26:22.119709 3586 consensus_queue.cc:1035] T 6c4417932190447ea1a9364f4014ba28 P a40e9d3ca975453296f44d142e052a7f [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d04a6257eb74f7f8ad11e16070dcf81" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 35323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:26:22.137331 3593 mvcc.cc:205] Tried to move back new op lower bound from 7316817437145305088 to 7316817436951142400. Current Snapshot: MvccSnapshot[applied={T|T < 7316817437145305088}]
I20260810 03:26:22.137816 3594 mvcc.cc:205] Tried to move back new op lower bound from 7316817437145305088 to 7316817436951142400. Current Snapshot: MvccSnapshot[applied={T|T < 7316817437145305088}]
I20260810 03:26:22.149165 2854 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:43362:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 03:26:22.149444 2854 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:43362:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 03:26:22.154011 2854 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b85e7fb7855e4c2282d7b0d2b3a71da9: Sending DeleteTablet for 3 replicas of tablet 6c4417932190447ea1a9364f4014ba28
I20260810 03:26:22.155236 3235 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c4417932190447ea1a9364f4014ba28 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:26:22 UTC) from {username='slave'} at 127.0.0.1:44790
I20260810 03:26:22.155293 3445 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c4417932190447ea1a9364f4014ba28 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:26:22 UTC) from {username='slave'} at 127.0.0.1:54964
I20260810 03:26:22.155781 3041 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c4417932190447ea1a9364f4014ba28 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:26:22 UTC) from {username='slave'} at 127.0.0.1:46910
I20260810 03:26:22.155898 3234 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c4417932190447ea1a9364f4014ba28 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:26:22 UTC) from {username='slave'} at 127.0.0.1:44790
I20260810 03:26:22.158519 28292 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iBH5/build/asan/bin/kudu pid 2944
I20260810 03:26:22.166178 3604 tablet_replica.cc:330] T 6c4417932190447ea1a9364f4014ba28 P 43dcd84a2dd742bbb7ee640b5638022b: stopping tablet replica
I20260810 03:26:22.166870 3604 raft_consensus.cc:2229] T 6c4417932190447ea1a9364f4014ba28 P 43dcd84a2dd742bbb7ee640b5638022b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 03:26:22.167577 3604 raft_consensus.cc:2258] T 6c4417932190447ea1a9364f4014ba28 P 43dcd84a2dd742bbb7ee640b5638022b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:26:22.169199 3607 tablet_replica.cc:330] T 6c4417932190447ea1a9364f4014ba28 P a40e9d3ca975453296f44d142e052a7f: stopping tablet replica
I20260810 03:26:22.169876 3604 ts_tablet_manager.cc:1905] T 6c4417932190447ea1a9364f4014ba28 P 43dcd84a2dd742bbb7ee640b5638022b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:26:22.170023 3607 raft_consensus.cc:2229] T 6c4417932190447ea1a9364f4014ba28 P a40e9d3ca975453296f44d142e052a7f [term 1 LEADER]: Raft consensus shutting down.
I20260810 03:26:22.170783 3607 raft_consensus.cc:2258] T 6c4417932190447ea1a9364f4014ba28 P a40e9d3ca975453296f44d142e052a7f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:26:22.173018 3607 ts_tablet_manager.cc:1905] T 6c4417932190447ea1a9364f4014ba28 P a40e9d3ca975453296f44d142e052a7f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:26:22.173231 3605 tablet_replica.cc:330] T 6c4417932190447ea1a9364f4014ba28 P 0d04a6257eb74f7f8ad11e16070dcf81: stopping tablet replica
I20260810 03:26:22.173695 3605 raft_consensus.cc:2229] T 6c4417932190447ea1a9364f4014ba28 P 0d04a6257eb74f7f8ad11e16070dcf81 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 03:26:22.176501 3567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:22.160010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46930 (local address 127.27.161.1:40381)
0810 03:26:22.160197 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0810 03:26:22.160203 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:26:22.173629 (+ 13426us) server_negotiation.cc:407] Connection header received
0810 03:26:22.174171 (+ 542us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:26:22.174177 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:26:22.174250 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:26:22.174331 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:26:22.175606 (+ 1275us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:26:22.175612 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:26:22.175617 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:26:22.175641 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:26:22.175681 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:26:22.175701 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:26:22.175828 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0810 03:26:22.176116 (+ 288us) server_negotiation.cc:299] Negotiation successful
0810 03:26:22.176182 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260810 03:26:22.177834 3136 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:26:22.178653 3605 raft_consensus.cc:2258] T 6c4417932190447ea1a9364f4014ba28 P 0d04a6257eb74f7f8ad11e16070dcf81 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:26:22.179356 3604 ts_tablet_manager.cc:1918] T 6c4417932190447ea1a9364f4014ba28 P 43dcd84a2dd742bbb7ee640b5638022b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 03:26:22.179667 3604 log.cc:1196] T 6c4417932190447ea1a9364f4014ba28 P 43dcd84a2dd742bbb7ee640b5638022b: Deleting WAL directory at /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786332336319544-28292-0/minicluster-data/ts-1/wal/wals/6c4417932190447ea1a9364f4014ba28
I20260810 03:26:22.180487 3604 ts_tablet_manager.cc:1939] T 6c4417932190447ea1a9364f4014ba28 P 43dcd84a2dd742bbb7ee640b5638022b: Deleting consensus metadata
I20260810 03:26:22.180929 3605 ts_tablet_manager.cc:1905] T 6c4417932190447ea1a9364f4014ba28 P 0d04a6257eb74f7f8ad11e16070dcf81: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:26:22.182873 2835 catalog_manager.cc:4841] TS 43dcd84a2dd742bbb7ee640b5638022b (127.27.161.2:41367): tablet 6c4417932190447ea1a9364f4014ba28 (table test-table [id=b96259704a4a4b8082c85dbf605835fd]) successfully deleted
W20260810 03:26:22.183531 2835 catalog_manager.cc:4808] TS 43dcd84a2dd742bbb7ee640b5638022b (127.27.161.2:41367): delete failed for tablet 6c4417932190447ea1a9364f4014ba28 because the tablet was not found. No further retry: Not found: Tablet not found: 6c4417932190447ea1a9364f4014ba28
I20260810 03:26:22.186587 3605 ts_tablet_manager.cc:1918] T 6c4417932190447ea1a9364f4014ba28 P 0d04a6257eb74f7f8ad11e16070dcf81: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 03:26:22.186733 3605 log.cc:1196] T 6c4417932190447ea1a9364f4014ba28 P 0d04a6257eb74f7f8ad11e16070dcf81: Deleting WAL directory at /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786332336319544-28292-0/minicluster-data/ts-2/wal/wals/6c4417932190447ea1a9364f4014ba28
I20260810 03:26:22.187178 3605 ts_tablet_manager.cc:1939] T 6c4417932190447ea1a9364f4014ba28 P 0d04a6257eb74f7f8ad11e16070dcf81: Deleting consensus metadata
I20260810 03:26:22.188767 2835 catalog_manager.cc:4841] TS 0d04a6257eb74f7f8ad11e16070dcf81 (127.27.161.3:35323): tablet 6c4417932190447ea1a9364f4014ba28 (table test-table [id=b96259704a4a4b8082c85dbf605835fd]) successfully deleted
I20260810 03:26:22.301913 28292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iBH5/build/asan/bin/kudu with pid 2944
W20260810 03:26:22.324784 2829 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.161.1:40381 (error 108)
I20260810 03:26:22.325093 28292 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iBH5/build/asan/bin/kudu pid 3149
I20260810 03:26:22.326622 3572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:22.326182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.1:40381 (local address 127.0.0.1:46932)
0810 03:26:22.326338 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0810 03:26:22.326483 (+ 145us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.1:40381: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260810 03:26:22.326862 2829 catalog_manager.cc:4584] TS a40e9d3ca975453296f44d142e052a7f (127.27.161.1:40381): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6c4417932190447ea1a9364f4014ba28: Network error: Client connection negotiation failed: client connection to 127.27.161.1:40381: connect: Connection refused (error 111)
I20260810 03:26:22.327852 3564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:22.325846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44804 (local address 127.27.161.2:41367)
0810 03:26:22.326060 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0810 03:26:22.326065 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:26:22.326083 (+ 18us) server_negotiation.cc:407] Connection header received
0810 03:26:22.326212 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:26:22.326217 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:26:22.326307 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:26:22.326408 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:26:22.327345 (+ 937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:26:22.327350 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:26:22.327353 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:26:22.327370 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:26:22.327403 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:26:22.327416 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:26:22.327516 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0810 03:26:22.327588 (+ 72us) server_negotiation.cc:299] Negotiation successful
0810 03:26:22.327649 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260810 03:26:22.328531 3333 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:26:22.380923 3572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:22.380340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.1:40381 (local address 127.0.0.1:46940)
0810 03:26:22.380594 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0810 03:26:22.380724 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.1:40381: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260810 03:26:22.382185 3572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:22.381724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.1:40381 (local address 127.0.0.1:46950)
0810 03:26:22.381898 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0810 03:26:22.382018 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.1:40381: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260810 03:26:22.417985 28292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iBH5/build/asan/bin/kudu with pid 3149
I20260810 03:26:22.441509 28292 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iBH5/build/asan/bin/kudu pid 3344
I20260810 03:26:22.443054 3572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:22.442254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.1:40381 (local address 127.0.0.1:46952)
0810 03:26:22.442727 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0810 03:26:22.442831 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.1:40381: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":360}
I20260810 03:26:22.444401 3572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:22.443906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.1:40381 (local address 127.0.0.1:46960)
0810 03:26:22.444124 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0810 03:26:22.444224 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.1:40381: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260810 03:26:22.444864 3569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:22.442381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54972 (local address 127.27.161.3:35323)
0810 03:26:22.442570 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0810 03:26:22.442575 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:26:22.442595 (+ 20us) server_negotiation.cc:407] Connection header received
0810 03:26:22.442747 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:26:22.442753 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:26:22.442861 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:26:22.442973 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:26:22.444090 (+ 1117us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:26:22.444098 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:26:22.444103 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:26:22.444126 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:26:22.444171 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:26:22.444190 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:26:22.444361 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0810 03:26:22.444475 (+ 114us) server_negotiation.cc:299] Negotiation successful
0810 03:26:22.444556 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260810 03:26:22.445595 3540 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:26:22.525216 3572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:22.524710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.1:40381 (local address 127.0.0.1:46976)
0810 03:26:22.524913 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0810 03:26:22.525028 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.1:40381: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260810 03:26:22.526436 3572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:22.525919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.1:40381 (local address 127.0.0.1:46980)
0810 03:26:22.526199 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0810 03:26:22.526291 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.1:40381: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155}
I20260810 03:26:22.536260 28292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iBH5/build/asan/bin/kudu with pid 3344
I20260810 03:26:22.566169 28292 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iBH5/build/asan/bin/kudu pid 2805
I20260810 03:26:22.567437 2901 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:26:22.677135 28292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iBH5/build/asan/bin/kudu with pid 2805
2026-08-10T03:26:22Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (2832 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260810 03:26:22.731519 28292 mini_oidc.cc:152] Starting JWKS server
I20260810 03:26:22.734519 28292 webserver.cc:460] Webserver started at https://127.0.0.1:33885/ using document root <none> and password file <none>
I20260810 03:26:22.734910 28292 mini_oidc.cc:189] Starting OIDC Discovery server
I20260810 03:26:22.735632 28292 webserver.cc:460] Webserver started at http://127.0.0.1:41385/ using document root <none> and password file <none>
2026-08-10T03:26:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T03:26:22Z Disabled control of system clock
I20260810 03:26:22.776176 28292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iBH5/build/asan/bin/kudu
/tmp/dist-test-task24iBH5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786332336319544-28292-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786332336319544-28292-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786332336319544-28292-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786332336319544-28292-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.161.62:33235
--webserver_interface=127.27.161.62
--webserver_port=0
--builtin_ntp_servers=127.27.161.20:36885
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.161.62:33235
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33885/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786332336319544-28292-0/testchainca.cert with env {}
W20260810 03:26:23.121409 3627 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:26:23.121763 3627 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:26:23.121901 3627 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:26:23.129447 3627 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:26:23.129602 3627 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:26:23.129673 3627 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:26:23.129752 3627 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:26:23.129814 3627 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260810 03:26:23.129904 3627 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260810 03:26:23.129974 3627 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:33885/jwks.json
I20260810 03:26:23.140971 3627 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.161.20:36885
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786332336319544-28292-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786332336319544-28292-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.161.62:33235
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.161.62:33235
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33885/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786332336319544-28292-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.161.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786332336319544-28292-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786332336319544-28292-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:26:23.142472 3627 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:26:23.144074 3627 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:26:23.156445 3634 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:26:23.156353 3635 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:26:23.156388 3638 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:26:23.157424 3627 server_base.cc:1034] running on GCE node
I20260810 03:26:23.158288 3627 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:26:23.159729 3627 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:26:23.160933 3627 hybrid_clock.cc:648] HybridClock initialized: now 1786332383160876 us; error 49 us; skew 500 ppm
I20260810 03:26:23.161304 3627 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 03:26:23.163643 3627 webserver.cc:460] Webserver started at http://127.27.161.62:33399/ using document root <none> and password file <none>
I20260810 03:26:23.164333 3627 fs_manager.cc:359] Metadata directory not provided
I20260810 03:26:23.164439 3627 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:26:23.164759 3627 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:26:23.167774 3627 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786332336319544-28292-0/minicluster-data/master-0/data/instance:
uuid: "459fdd36a97c4a878561ae8347398867"
format_stamp: "Formatted at 2026-08-10 03:26:23 on dist-test-slave-vpvm"
server_key: "d72454069a72d191b36b34167e3044b5"
server_key_iv: "37b4bc7ff982931072d2bcf1216292a8"
server_key_version: "encryptionkey@0"
I20260810 03:26:23.168576 3627 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786332336319544-28292-0/minicluster-data/master-0/wal/instance:
uuid: "459fdd36a97c4a878561ae8347398867"
format_stamp: "Formatted at 2026-08-10 03:26:23 on dist-test-slave-vpvm"
server_key: "d72454069a72d191b36b34167e3044b5"
server_key_iv: "37b4bc7ff982931072d2bcf1216292a8"
server_key_version: "encryptionkey@0"
I20260810 03:26:23.173921 3627 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260810 03:26:23.177700 3653 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:26:23.179283 3627 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 03:26:23.179473 3627 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786332336319544-28292-0/minicluster-data/master-0/data,/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786332336319544-28292-0/minicluster-data/master-0/wal
uuid: "459fdd36a97c4a878561ae8347398867"
format_stamp: "Formatted at 2026-08-10 03:26:23 on dist-test-slave-vpvm"
server_key: "d72454069a72d191b36b34167e3044b5"
server_key_iv: "37b4bc7ff982931072d2bcf1216292a8"
server_key_version: "encryptionkey@0"
I20260810 03:26:23.179710 3627 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786332336319544-28292-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786332336319544-28292-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786332336319544-28292-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:26:23.203601 3627 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:26:23.204531 3627 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:26:23.204840 3627 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:26:23.223379 3627 rpc_server.cc:307] RPC server started. Bound to: 127.27.161.62:33235
I20260810 03:26:23.223429 3734 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.161.62:33235 every 8 connection(s)
I20260810 03:26:23.224781 3627 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786332336319544-28292-0/minicluster-data/master-0/data/info.pb
I20260810 03:26:23.228916 3735 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:26:23.232570 28292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iBH5/build/asan/bin/kudu as pid 3627
I20260810 03:26:23.232777 28292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786332336319544-28292-0/minicluster-data/master-0/wal/instance
I20260810 03:26:23.233609 28292 external_mini_cluster.cc:1428] Setting key fd0e7e2cb058fbbb99411e3c541a6e9f
I20260810 03:26:23.242041 3735 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 459fdd36a97c4a878561ae8347398867: Bootstrap starting.
I20260810 03:26:23.245298 3735 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 459fdd36a97c4a878561ae8347398867: Neither blocks nor log segments found. Creating new log.
I20260810 03:26:23.246487 3735 log.cc:824] T 00000000000000000000000000000000 P 459fdd36a97c4a878561ae8347398867: Log is configured to *not* fsync() on all Append() calls
I20260810 03:26:23.247279 3740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:23.235718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56338 (local address 127.27.161.62:33235)
0810 03:26:23.236757 (+ 1039us) server_negotiation.cc:206] Beginning negotiation
0810 03:26:23.236768 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 03:26:23.236811 (+ 43us) server_negotiation.cc:407] Connection header received
0810 03:26:23.238491 (+ 1680us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:26:23.238518 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:26:23.239047 (+ 529us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:26:23.239447 (+ 400us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:26:23.240048 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:23.241342 (+ 1294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:23.242771 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:23.243483 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:23.243672 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:23.244811 (+ 1139us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:26:23.244819 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:26:23.244834 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:26:23.245399 (+ 565us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:26:23.245514 (+ 115us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:26:23.245542 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:26:23.245667 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0810 03:26:23.245886 (+ 219us) server_negotiation.cc:299] Negotiation successful
0810 03:26:23.246230 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":627,"thread_start_us":398,"threads_started":1}
I20260810 03:26:23.249402 3735 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 459fdd36a97c4a878561ae8347398867: No bootstrap required, opened a new log
I20260810 03:26:23.251053 28292 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 03:26:23.256989 3735 raft_consensus.cc:348] T 00000000000000000000000000000000 P 459fdd36a97c4a878561ae8347398867 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "459fdd36a97c4a878561ae8347398867" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 33235 } }
I20260810 03:26:23.257359 3735 raft_consensus.cc:374] T 00000000000000000000000000000000 P 459fdd36a97c4a878561ae8347398867 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:26:23.257450 3735 raft_consensus.cc:729] T 00000000000000000000000000000000 P 459fdd36a97c4a878561ae8347398867 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 459fdd36a97c4a878561ae8347398867, State: Initialized, Role: FOLLOWER
I20260810 03:26:23.258028 3735 consensus_queue.cc:260] T 00000000000000000000000000000000 P 459fdd36a97c4a878561ae8347398867 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "459fdd36a97c4a878561ae8347398867" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 33235 } }
I20260810 03:26:23.258249 3735 raft_consensus.cc:388] T 00000000000000000000000000000000 P 459fdd36a97c4a878561ae8347398867 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 03:26:23.258373 3735 raft_consensus.cc:482] T 00000000000000000000000000000000 P 459fdd36a97c4a878561ae8347398867 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 03:26:23.258533 3735 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 459fdd36a97c4a878561ae8347398867 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:26:23.261044 3735 raft_consensus.cc:504] T 00000000000000000000000000000000 P 459fdd36a97c4a878561ae8347398867 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "459fdd36a97c4a878561ae8347398867" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 33235 } }
I20260810 03:26:23.261597 3735 leader_election.cc:302] T 00000000000000000000000000000000 P 459fdd36a97c4a878561ae8347398867 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 459fdd36a97c4a878561ae8347398867; no voters:
I20260810 03:26:23.262257 3735 leader_election.cc:288] T 00000000000000000000000000000000 P 459fdd36a97c4a878561ae8347398867 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 03:26:23.262435 3743 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 459fdd36a97c4a878561ae8347398867 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:26:23.263212 3743 raft_consensus.cc:686] T 00000000000000000000000000000000 P 459fdd36a97c4a878561ae8347398867 [term 1 LEADER]: Becoming Leader. State: Replica: 459fdd36a97c4a878561ae8347398867, State: Running, Role: LEADER
I20260810 03:26:23.263839 3743 consensus_queue.cc:237] T 00000000000000000000000000000000 P 459fdd36a97c4a878561ae8347398867 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "459fdd36a97c4a878561ae8347398867" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 33235 } }
I20260810 03:26:23.264411 3735 sys_catalog.cc:564] T 00000000000000000000000000000000 P 459fdd36a97c4a878561ae8347398867 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:26:23.266776 3745 sys_catalog.cc:455] T 00000000000000000000000000000000 P 459fdd36a97c4a878561ae8347398867 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 459fdd36a97c4a878561ae8347398867. Latest consensus state: current_term: 1 leader_uuid: "459fdd36a97c4a878561ae8347398867" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "459fdd36a97c4a878561ae8347398867" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 33235 } } }
I20260810 03:26:23.267275 3745 sys_catalog.cc:458] T 00000000000000000000000000000000 P 459fdd36a97c4a878561ae8347398867 [sys.catalog]: This master's current role is: LEADER
I20260810 03:26:23.266798 3744 sys_catalog.cc:455] T 00000000000000000000000000000000 P 459fdd36a97c4a878561ae8347398867 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "459fdd36a97c4a878561ae8347398867" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "459fdd36a97c4a878561ae8347398867" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 33235 } } }
I20260810 03:26:23.268378 3744 sys_catalog.cc:458] T 00000000000000000000000000000000 P 459fdd36a97c4a878561ae8347398867 [sys.catalog]: This master's current role is: LEADER
I20260810 03:26:23.274724 3750 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 03:26:23.280476 3750 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 03:26:23.291349 3750 catalog_manager.cc:1347] Generated new cluster ID: 9cdaa5dc23864bd8a4824ef3ca04412a
I20260810 03:26:23.291496 3750 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 03:26:23.302165 3750 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 03:26:23.303668 3750 catalog_manager.cc:1504] Loading token signing keys...
I20260810 03:26:23.313482 3750 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 459fdd36a97c4a878561ae8347398867: Generated new TSK 0
I20260810 03:26:23.314757 3750 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260810 03:26:23.366971 3781 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260810 03:26:23.367202 3781 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260810 03:26:23.368515 3740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:23.341216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56348 (local address 127.27.161.62:33235)
0810 03:26:23.341460 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0810 03:26:23.341468 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 03:26:23.341990 (+ 522us) server_negotiation.cc:407] Connection header received
0810 03:26:23.342443 (+ 453us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:26:23.342448 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:26:23.342532 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:26:23.342656 (+ 124us) server_negotiation.cc:226] Negotiated authn=JWT
0810 03:26:23.343448 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:23.344730 (+ 1282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:23.347041 (+ 2311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:23.347507 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:23.347617 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:23.348413 (+ 796us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0810 03:26:23.367760 (+ 19347us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33885/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0810 03:26:23.368198 (+ 438us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:56348: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33885/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":78}
W20260810 03:26:23.368647 3740 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:56348: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33885/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260810 03:26:23.370303 3777 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:26:23.341093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:33235 (local address 127.0.0.1:56348)
0810 03:26:23.341834 (+ 741us) negotiation.cc:107] Waiting for socket to connect
0810 03:26:23.341855 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0810 03:26:23.342105 (+ 250us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:26:23.342730 (+ 625us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:26:23.342736 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:26:23.342750 (+ 14us) client_negotiation.cc:189] Negotiated authn=JWT
0810 03:26:23.343157 (+ 407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:26:23.343170 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:26:23.344932 (+ 1762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:26:23.344938 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:26:23.346823 (+ 1885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:26:23.346842 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:26:23.347712 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:26:23.347720 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:26:23.347933 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:23.348040 (+ 107us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0810 03:26:23.368084 (+ 20044us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33885/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0810 03:26:23.368406 (+ 322us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.27.161.62:33235: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33885/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":535,"thread_start_us":349,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.27.161.62:33235: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33885/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260810 03:26:23.371973 28292 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iBH5/build/asan/bin/kudu pid 3627
I20260810 03:26:23.373342 3727 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:26:23.472975 28292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iBH5/build/asan/bin/kudu with pid 3627
2026-08-10T03:26:23Z chronyd exiting
I20260810 03:26:23.536087 28292 test_util.cc:175] -----------------------------------------------
I20260810 03:26:23.536262 28292 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786332336319544-28292-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (809 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260810 03:26:23.541560 28292 mini_oidc.cc:152] Starting JWKS server
I20260810 03:26:23.543584 28292 webserver.cc:460] Webserver started at https://127.0.0.1:43433/ using document root <none> and password file <none>
I20260810 03:26:23.543826 28292 mini_oidc.cc:189] Starting OIDC Discovery server
I20260810 03:26:23.544538 28292 webserver.cc:460] Webserver started at http://127.0.0.1:45507/ using document root <none> and password file <none>
2026-08-10T03:26:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T03:26:23Z Disabled control of system clock
I20260810 03:26:23.585487 28292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iBH5/build/asan/bin/kudu
/tmp/dist-test-task24iBH5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786332336319544-28292-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786332336319544-28292-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786332336319544-28292-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786332336319544-28292-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.161.62:44997
--webserver_interface=127.27.161.62
--webserver_port=0
--builtin_ntp_servers=127.27.161.20:35383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.161.62:44997
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43433/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786332336319544-28292-0/testchainca.cert with env {}
W20260810 03:26:23.948874 3789 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:26:23.949220 3789 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:26:23.949360 3789 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:26:23.956835 3789 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:26:23.956974 3789 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:26:23.957047 3789 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:26:23.957125 3789 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:26:23.957190 3789 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260810 03:26:23.957278 3789 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260810 03:26:23.957340 3789 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:43433/jwks.json
I20260810 03:26:23.968168 3789 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.161.20:35383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786332336319544-28292-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786332336319544-28292-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.161.62:44997
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.161.62:44997
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43433/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786332336319544-28292-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.161.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786332336319544-28292-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786332336319544-28292-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:26:23.969735 3789 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:26:23.971172 3789 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:26:23.983596 3803 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:26:23.983672 3796 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:26:23.983642 3800 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:26:23.984434 3789 server_base.cc:1034] running on GCE node
I20260810 03:26:23.985397 3789 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:26:23.986976 3789 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:26:23.988219 3789 hybrid_clock.cc:648] HybridClock initialized: now 1786332383988156 us; error 47 us; skew 500 ppm
I20260810 03:26:23.988622 3789 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 03:26:23.991168 3789 webserver.cc:460] Webserver started at http://127.27.161.62:35283/ using document root <none> and password file <none>
I20260810 03:26:23.992066 3789 fs_manager.cc:359] Metadata directory not provided
I20260810 03:26:23.992195 3789 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:26:23.992539 3789 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:26:23.995793 3789 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786332336319544-28292-0/minicluster-data/master-0/data/instance:
uuid: "a729fb8a61c948d893fddc214a3bdcb6"
format_stamp: "Formatted at 2026-08-10 03:26:23 on dist-test-slave-vpvm"
server_key: "218cb702a8562eb9297adfb0be8b444f"
server_key_iv: "4e464ade36163e2847ad6f6c4cac2ade"
server_key_version: "encryptionkey@0"
I20260810 03:26:23.996660 3789 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786332336319544-28292-0/minicluster-data/master-0/wal/instance:
uuid: "a729fb8a61c948d893fddc214a3bdcb6"
format_stamp: "Formatted at 2026-08-10 03:26:23 on dist-test-slave-vpvm"
server_key: "218cb702a8562eb9297adfb0be8b444f"
server_key_iv: "4e464ade36163e2847ad6f6c4cac2ade"
server_key_version: "encryptionkey@0"
I20260810 03:26:24.002808 3789 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.003s
I20260810 03:26:24.007238 3818 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:26:24.008780 3789 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.005s sys 0.000s
I20260810 03:26:24.008981 3789 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786332336319544-28292-0/minicluster-data/master-0/data,/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786332336319544-28292-0/minicluster-data/master-0/wal
uuid: "a729fb8a61c948d893fddc214a3bdcb6"
format_stamp: "Formatted at 2026-08-10 03:26:23 on dist-test-slave-vpvm"
server_key: "218cb702a8562eb9297adfb0be8b444f"
server_key_iv: "4e464ade36163e2847ad6f6c4cac2ade"
server_key_version: "encryptionkey@0"
I20260810 03:26:24.009223 3789 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786332336319544-28292-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786332336319544-28292-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786332336319544-28292-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:26:24.033290 3789 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:26:24.034341 3789 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:26:24.034816 3789 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:26:24.054219 3789 rpc_server.cc:307] RPC server started. Bound to: 127.27.161.62:44997
I20260810 03:26:24.054237 3893 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.161.62:44997 every 8 connection(s)
I20260810 03:26:24.055774 3789 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786332336319544-28292-0/minicluster-data/master-0/data/info.pb
I20260810 03:26:24.059749 3894 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:26:24.062701 28292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iBH5/build/asan/bin/kudu as pid 3789
I20260810 03:26:24.062918 28292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786332336319544-28292-0/minicluster-data/master-0/wal/instance
I20260810 03:26:24.063764 28292 external_mini_cluster.cc:1428] Setting key 0ba69d28827c04930350f59a94a16e65
I20260810 03:26:24.072430 3894 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a729fb8a61c948d893fddc214a3bdcb6: Bootstrap starting.
I20260810 03:26:24.076136 3894 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a729fb8a61c948d893fddc214a3bdcb6: Neither blocks nor log segments found. Creating new log.
I20260810 03:26:24.077785 3894 log.cc:824] T 00000000000000000000000000000000 P a729fb8a61c948d893fddc214a3bdcb6: Log is configured to *not* fsync() on all Append() calls
I20260810 03:26:24.077847 3901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:24.066004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37720 (local address 127.27.161.62:44997)
0810 03:26:24.067151 (+ 1147us) server_negotiation.cc:206] Beginning negotiation
0810 03:26:24.067164 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 03:26:24.067210 (+ 46us) server_negotiation.cc:407] Connection header received
0810 03:26:24.068901 (+ 1691us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:26:24.068925 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:26:24.069432 (+ 507us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:26:24.069830 (+ 398us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:26:24.070469 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:24.071748 (+ 1279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:24.073078 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:24.073762 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:24.073946 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:24.075095 (+ 1149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:26:24.075103 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:26:24.075119 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:26:24.075788 (+ 669us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:26:24.075893 (+ 105us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:26:24.075916 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:26:24.076065 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0810 03:26:24.076307 (+ 242us) server_negotiation.cc:299] Negotiation successful
0810 03:26:24.076730 (+ 423us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":682,"thread_start_us":424,"threads_started":1}
I20260810 03:26:24.081830 3894 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a729fb8a61c948d893fddc214a3bdcb6: No bootstrap required, opened a new log
I20260810 03:26:24.082864 28292 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 03:26:24.089890 3894 raft_consensus.cc:348] T 00000000000000000000000000000000 P a729fb8a61c948d893fddc214a3bdcb6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a729fb8a61c948d893fddc214a3bdcb6" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 44997 } }
I20260810 03:26:24.090431 3894 raft_consensus.cc:374] T 00000000000000000000000000000000 P a729fb8a61c948d893fddc214a3bdcb6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:26:24.090546 3894 raft_consensus.cc:729] T 00000000000000000000000000000000 P a729fb8a61c948d893fddc214a3bdcb6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a729fb8a61c948d893fddc214a3bdcb6, State: Initialized, Role: FOLLOWER
I20260810 03:26:24.091199 3894 consensus_queue.cc:260] T 00000000000000000000000000000000 P a729fb8a61c948d893fddc214a3bdcb6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a729fb8a61c948d893fddc214a3bdcb6" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 44997 } }
I20260810 03:26:24.091460 3894 raft_consensus.cc:388] T 00000000000000000000000000000000 P a729fb8a61c948d893fddc214a3bdcb6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 03:26:24.091602 3894 raft_consensus.cc:482] T 00000000000000000000000000000000 P a729fb8a61c948d893fddc214a3bdcb6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 03:26:24.091769 3894 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a729fb8a61c948d893fddc214a3bdcb6 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:26:24.094302 3894 raft_consensus.cc:504] T 00000000000000000000000000000000 P a729fb8a61c948d893fddc214a3bdcb6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a729fb8a61c948d893fddc214a3bdcb6" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 44997 } }
I20260810 03:26:24.094846 3894 leader_election.cc:302] T 00000000000000000000000000000000 P a729fb8a61c948d893fddc214a3bdcb6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a729fb8a61c948d893fddc214a3bdcb6; no voters:
I20260810 03:26:24.095458 3894 leader_election.cc:288] T 00000000000000000000000000000000 P a729fb8a61c948d893fddc214a3bdcb6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 03:26:24.095633 3906 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a729fb8a61c948d893fddc214a3bdcb6 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:26:24.096360 3906 raft_consensus.cc:686] T 00000000000000000000000000000000 P a729fb8a61c948d893fddc214a3bdcb6 [term 1 LEADER]: Becoming Leader. State: Replica: a729fb8a61c948d893fddc214a3bdcb6, State: Running, Role: LEADER
I20260810 03:26:24.096948 3906 consensus_queue.cc:237] T 00000000000000000000000000000000 P a729fb8a61c948d893fddc214a3bdcb6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a729fb8a61c948d893fddc214a3bdcb6" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 44997 } }
I20260810 03:26:24.097584 3894 sys_catalog.cc:564] T 00000000000000000000000000000000 P a729fb8a61c948d893fddc214a3bdcb6 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:26:24.099680 3909 sys_catalog.cc:455] T 00000000000000000000000000000000 P a729fb8a61c948d893fddc214a3bdcb6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a729fb8a61c948d893fddc214a3bdcb6. Latest consensus state: current_term: 1 leader_uuid: "a729fb8a61c948d893fddc214a3bdcb6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a729fb8a61c948d893fddc214a3bdcb6" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 44997 } } }
I20260810 03:26:24.100150 3909 sys_catalog.cc:458] T 00000000000000000000000000000000 P a729fb8a61c948d893fddc214a3bdcb6 [sys.catalog]: This master's current role is: LEADER
I20260810 03:26:24.100109 3908 sys_catalog.cc:455] T 00000000000000000000000000000000 P a729fb8a61c948d893fddc214a3bdcb6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a729fb8a61c948d893fddc214a3bdcb6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a729fb8a61c948d893fddc214a3bdcb6" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 44997 } } }
I20260810 03:26:24.100461 3908 sys_catalog.cc:458] T 00000000000000000000000000000000 P a729fb8a61c948d893fddc214a3bdcb6 [sys.catalog]: This master's current role is: LEADER
I20260810 03:26:24.103066 3919 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 03:26:24.109323 3919 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 03:26:24.119216 3919 catalog_manager.cc:1347] Generated new cluster ID: f47c45043fc24f3fb64197e1b580ca86
I20260810 03:26:24.119352 3919 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 03:26:24.129074 3919 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 03:26:24.131044 3919 catalog_manager.cc:1504] Loading token signing keys...
I20260810 03:26:24.146685 3919 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a729fb8a61c948d893fddc214a3bdcb6: Generated new TSK 0
I20260810 03:26:24.147882 3919 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260810 03:26:24.183001 3942 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260810 03:26:24.183164 3942 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260810 03:26:24.184422 3901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:24.164226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37722 (local address 127.27.161.62:44997)
0810 03:26:24.164425 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0810 03:26:24.164433 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 03:26:24.164868 (+ 435us) server_negotiation.cc:407] Connection header received
0810 03:26:24.165107 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:26:24.165111 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:26:24.165188 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:26:24.165335 (+ 147us) server_negotiation.cc:226] Negotiated authn=JWT
0810 03:26:24.165865 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:24.166693 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:24.168106 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:24.168593 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:24.168738 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:24.169336 (+ 598us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0810 03:26:24.183679 (+ 14343us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43433/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0810 03:26:24.184108 (+ 429us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:37722: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43433/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":57}
W20260810 03:26:24.184598 3901 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:37722: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43433/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260810 03:26:24.184670 3941 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:26:24.164122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:44997 (local address 127.0.0.1:37722)
0810 03:26:24.164750 (+ 628us) negotiation.cc:107] Waiting for socket to connect
0810 03:26:24.164770 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 03:26:24.164929 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:26:24.165371 (+ 442us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:26:24.165376 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:26:24.165390 (+ 14us) client_negotiation.cc:189] Negotiated authn=JWT
0810 03:26:24.165693 (+ 303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:26:24.165702 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:26:24.166844 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:26:24.166848 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:26:24.167932 (+ 1084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:26:24.167944 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:26:24.168787 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:26:24.168794 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:26:24.168960 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:24.169003 (+ 43us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0810 03:26:24.183971 (+ 14968us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43433/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0810 03:26:24.184271 (+ 300us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.27.161.62:44997: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43433/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":507,"thread_start_us":339,"threads_started":1}
I20260810 03:26:24.185983 28292 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iBH5/build/asan/bin/kudu pid 3789
I20260810 03:26:24.187435 3887 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:26:24.288789 28292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iBH5/build/asan/bin/kudu with pid 3789
2026-08-10T03:26:24Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (810 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260810 03:26:24.350521 28292 mini_oidc.cc:152] Starting JWKS server
I20260810 03:26:24.352535 28292 webserver.cc:460] Webserver started at https://127.0.0.1:45167/ using document root <none> and password file <none>
I20260810 03:26:24.352787 28292 mini_oidc.cc:189] Starting OIDC Discovery server
I20260810 03:26:24.353451 28292 webserver.cc:460] Webserver started at http://127.0.0.1:41387/ using document root <none> and password file <none>
2026-08-10T03:26:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T03:26:24Z Disabled control of system clock
I20260810 03:26:24.392855 28292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iBH5/build/asan/bin/kudu
/tmp/dist-test-task24iBH5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786332336319544-28292-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786332336319544-28292-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786332336319544-28292-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786332336319544-28292-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.161.62:38651
--webserver_interface=127.27.161.62
--webserver_port=0
--builtin_ntp_servers=127.27.161.20:34319
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.161.62:38651
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45167/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260810 03:26:24.746659 3956 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:26:24.746991 3956 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:26:24.747115 3956 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:26:24.754594 3956 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:26:24.754709 3956 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:26:24.754748 3956 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:26:24.754830 3956 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:26:24.754884 3956 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260810 03:26:24.754987 3956 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260810 03:26:24.755044 3956 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:45167/jwks.json
I20260810 03:26:24.764923 3956 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.161.20:34319
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786332336319544-28292-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786332336319544-28292-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.161.62:38651
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.161.62:38651
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45167/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786332336319544-28292-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.161.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786332336319544-28292-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:26:24.766331 3956 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:26:24.768205 3956 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:26:24.780143 3963 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:26:24.780105 3967 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:26:24.780159 3964 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:26:24.780979 3956 server_base.cc:1034] running on GCE node
I20260810 03:26:24.781822 3956 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:26:24.783397 3956 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:26:24.784634 3956 hybrid_clock.cc:648] HybridClock initialized: now 1786332384784572 us; error 53 us; skew 500 ppm
I20260810 03:26:24.785008 3956 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 03:26:24.787411 3956 webserver.cc:460] Webserver started at http://127.27.161.62:42237/ using document root <none> and password file <none>
I20260810 03:26:24.788146 3956 fs_manager.cc:359] Metadata directory not provided
I20260810 03:26:24.788242 3956 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:26:24.788579 3956 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:26:24.791648 3956 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786332336319544-28292-0/minicluster-data/master-0/data/instance:
uuid: "589ab95f927843fbb288795f5293b6c9"
format_stamp: "Formatted at 2026-08-10 03:26:24 on dist-test-slave-vpvm"
server_key: "1df7ee78eba7dfc58b9737c3646efc84"
server_key_iv: "a779cc5ef1e1c607b023dde0d2dcd46a"
server_key_version: "encryptionkey@0"
I20260810 03:26:24.792497 3956 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786332336319544-28292-0/minicluster-data/master-0/wal/instance:
uuid: "589ab95f927843fbb288795f5293b6c9"
format_stamp: "Formatted at 2026-08-10 03:26:24 on dist-test-slave-vpvm"
server_key: "1df7ee78eba7dfc58b9737c3646efc84"
server_key_iv: "a779cc5ef1e1c607b023dde0d2dcd46a"
server_key_version: "encryptionkey@0"
I20260810 03:26:24.798094 3956 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260810 03:26:24.801921 3973 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:26:24.803510 3956 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 03:26:24.803715 3956 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786332336319544-28292-0/minicluster-data/master-0/data,/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786332336319544-28292-0/minicluster-data/master-0/wal
uuid: "589ab95f927843fbb288795f5293b6c9"
format_stamp: "Formatted at 2026-08-10 03:26:24 on dist-test-slave-vpvm"
server_key: "1df7ee78eba7dfc58b9737c3646efc84"
server_key_iv: "a779cc5ef1e1c607b023dde0d2dcd46a"
server_key_version: "encryptionkey@0"
I20260810 03:26:24.803946 3956 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786332336319544-28292-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786332336319544-28292-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786332336319544-28292-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:26:24.830343 3956 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:26:24.831285 3956 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:26:24.831610 3956 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:26:24.851127 3956 rpc_server.cc:307] RPC server started. Bound to: 127.27.161.62:38651
I20260810 03:26:24.851133 4052 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.161.62:38651 every 8 connection(s)
I20260810 03:26:24.852525 3956 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786332336319544-28292-0/minicluster-data/master-0/data/info.pb
I20260810 03:26:24.856524 4053 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:26:24.859853 28292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iBH5/build/asan/bin/kudu as pid 3956
I20260810 03:26:24.860086 28292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786332336319544-28292-0/minicluster-data/master-0/wal/instance
I20260810 03:26:24.860878 28292 external_mini_cluster.cc:1428] Setting key 37ddc452c18df5efa1bd1de94e44d6ae
I20260810 03:26:24.869889 4053 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 589ab95f927843fbb288795f5293b6c9: Bootstrap starting.
I20260810 03:26:24.873258 4053 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 589ab95f927843fbb288795f5293b6c9: Neither blocks nor log segments found. Creating new log.
I20260810 03:26:24.874504 4053 log.cc:824] T 00000000000000000000000000000000 P 589ab95f927843fbb288795f5293b6c9: Log is configured to *not* fsync() on all Append() calls
I20260810 03:26:24.875494 4059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:24.863125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56384 (local address 127.27.161.62:38651)
0810 03:26:24.864378 (+ 1253us) server_negotiation.cc:206] Beginning negotiation
0810 03:26:24.864391 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 03:26:24.864438 (+ 47us) server_negotiation.cc:407] Connection header received
0810 03:26:24.866232 (+ 1794us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:26:24.866260 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:26:24.866835 (+ 575us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:26:24.867276 (+ 441us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:26:24.867920 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:24.869248 (+ 1328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:24.870749 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:24.871650 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:24.871874 (+ 224us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:24.872997 (+ 1123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:26:24.873005 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:26:24.873023 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:26:24.873620 (+ 597us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:26:24.873733 (+ 113us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:26:24.873757 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:26:24.873859 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0810 03:26:24.874147 (+ 288us) server_negotiation.cc:299] Negotiation successful
0810 03:26:24.874467 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":805,"thread_start_us":566,"threads_started":1}
I20260810 03:26:24.877480 4053 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 589ab95f927843fbb288795f5293b6c9: No bootstrap required, opened a new log
I20260810 03:26:24.879294 28292 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 03:26:24.885406 4053 raft_consensus.cc:348] T 00000000000000000000000000000000 P 589ab95f927843fbb288795f5293b6c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "589ab95f927843fbb288795f5293b6c9" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 38651 } }
I20260810 03:26:24.885807 4053 raft_consensus.cc:374] T 00000000000000000000000000000000 P 589ab95f927843fbb288795f5293b6c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:26:24.885900 4053 raft_consensus.cc:729] T 00000000000000000000000000000000 P 589ab95f927843fbb288795f5293b6c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 589ab95f927843fbb288795f5293b6c9, State: Initialized, Role: FOLLOWER
I20260810 03:26:24.886603 4053 consensus_queue.cc:260] T 00000000000000000000000000000000 P 589ab95f927843fbb288795f5293b6c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "589ab95f927843fbb288795f5293b6c9" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 38651 } }
I20260810 03:26:24.886869 4053 raft_consensus.cc:388] T 00000000000000000000000000000000 P 589ab95f927843fbb288795f5293b6c9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 03:26:24.887027 4053 raft_consensus.cc:482] T 00000000000000000000000000000000 P 589ab95f927843fbb288795f5293b6c9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 03:26:24.887200 4053 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 589ab95f927843fbb288795f5293b6c9 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:26:24.889391 4053 raft_consensus.cc:504] T 00000000000000000000000000000000 P 589ab95f927843fbb288795f5293b6c9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "589ab95f927843fbb288795f5293b6c9" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 38651 } }
I20260810 03:26:24.889901 4053 leader_election.cc:302] T 00000000000000000000000000000000 P 589ab95f927843fbb288795f5293b6c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 589ab95f927843fbb288795f5293b6c9; no voters:
I20260810 03:26:24.890556 4053 leader_election.cc:288] T 00000000000000000000000000000000 P 589ab95f927843fbb288795f5293b6c9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 03:26:24.890702 4062 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 589ab95f927843fbb288795f5293b6c9 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:26:24.891723 4062 raft_consensus.cc:686] T 00000000000000000000000000000000 P 589ab95f927843fbb288795f5293b6c9 [term 1 LEADER]: Becoming Leader. State: Replica: 589ab95f927843fbb288795f5293b6c9, State: Running, Role: LEADER
I20260810 03:26:24.892313 4062 consensus_queue.cc:237] T 00000000000000000000000000000000 P 589ab95f927843fbb288795f5293b6c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "589ab95f927843fbb288795f5293b6c9" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 38651 } }
I20260810 03:26:24.892915 4053 sys_catalog.cc:564] T 00000000000000000000000000000000 P 589ab95f927843fbb288795f5293b6c9 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:26:24.895035 4065 sys_catalog.cc:455] T 00000000000000000000000000000000 P 589ab95f927843fbb288795f5293b6c9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "589ab95f927843fbb288795f5293b6c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "589ab95f927843fbb288795f5293b6c9" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 38651 } } }
I20260810 03:26:24.895356 4065 sys_catalog.cc:458] T 00000000000000000000000000000000 P 589ab95f927843fbb288795f5293b6c9 [sys.catalog]: This master's current role is: LEADER
I20260810 03:26:24.895525 4066 sys_catalog.cc:455] T 00000000000000000000000000000000 P 589ab95f927843fbb288795f5293b6c9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 589ab95f927843fbb288795f5293b6c9. Latest consensus state: current_term: 1 leader_uuid: "589ab95f927843fbb288795f5293b6c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "589ab95f927843fbb288795f5293b6c9" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 38651 } } }
I20260810 03:26:24.895893 4066 sys_catalog.cc:458] T 00000000000000000000000000000000 P 589ab95f927843fbb288795f5293b6c9 [sys.catalog]: This master's current role is: LEADER
I20260810 03:26:24.898047 4072 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 03:26:24.903560 4072 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 03:26:24.913760 4072 catalog_manager.cc:1347] Generated new cluster ID: ce85644a4fd641049dafdd7ec62e10b2
I20260810 03:26:24.913916 4072 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 03:26:24.940999 4072 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 03:26:24.942497 4072 catalog_manager.cc:1504] Loading token signing keys...
I20260810 03:26:24.952730 4072 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 589ab95f927843fbb288795f5293b6c9: Generated new TSK 0
I20260810 03:26:24.953697 4072 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 03:26:24.979473 4059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:24.970708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56398 (local address 127.27.161.62:38651)
0810 03:26:24.970926 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0810 03:26:24.970932 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:26:24.971357 (+ 425us) server_negotiation.cc:407] Connection header received
0810 03:26:24.971606 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:26:24.971610 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:26:24.971687 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:26:24.971834 (+ 147us) server_negotiation.cc:226] Negotiated authn=JWT
0810 03:26:24.972391 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:24.973137 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:24.975208 (+ 2071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:24.975679 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:24.975792 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:24.976459 (+ 667us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0810 03:26:24.978817 (+ 2358us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45167/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0810 03:26:24.979171 (+ 354us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:56398: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45167/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":58}
W20260810 03:26:24.979622 4059 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:56398: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45167/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260810 03:26:24.981688 4100 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:26:24.970511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:38651 (local address 127.0.0.1:56398)
0810 03:26:24.971199 (+ 688us) negotiation.cc:107] Waiting for socket to connect
0810 03:26:24.971220 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0810 03:26:24.971394 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:26:24.971869 (+ 475us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:26:24.971874 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:26:24.971888 (+ 14us) client_negotiation.cc:189] Negotiated authn=JWT
0810 03:26:24.972237 (+ 349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:26:24.972250 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:26:24.973270 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:26:24.973277 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:26:24.975018 (+ 1741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:26:24.975035 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:26:24.975889 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:26:24.975897 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:26:24.976088 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:24.976153 (+ 65us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0810 03:26:24.979048 (+ 2895us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45167/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0810 03:26:24.981226 (+ 2178us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.27.161.62:38651: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45167/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":533,"mutex_wait_us":1714,"thread_start_us":335,"threads_started":1}
I20260810 03:26:24.983153 28292 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iBH5/build/asan/bin/kudu pid 3956
I20260810 03:26:24.984539 4047 generic_service.cc:196] Checking for leaks (request via RPC)
E20260810 03:26:24.993932 4102 webserver.cc:533] Webserver: sslize failed:
E20260810 03:26:24.994112 4102 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260810 03:26:25.081167 28292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iBH5/build/asan/bin/kudu with pid 3956
2026-08-10T03:26:25Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (794 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-08-10T03:26:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T03:26:25Z Disabled control of system clock
I20260810 03:26:25.183789 28292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iBH5/build/asan/bin/kudu
/tmp/dist-test-task24iBH5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786332336319544-28292-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786332336319544-28292-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786332336319544-28292-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786332336319544-28292-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.161.62:41823
--webserver_interface=127.27.161.62
--webserver_port=0
--builtin_ntp_servers=127.27.161.20:37811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.161.62:41823
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786332336319544-28292-0/testchainca.cert with env {}
W20260810 03:26:25.535561 4108 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:26:25.535910 4108 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:26:25.536048 4108 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:26:25.543690 4108 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:26:25.543821 4108 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:26:25.543864 4108 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:26:25.543947 4108 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:26:25.544004 4108 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260810 03:26:25.544094 4108 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260810 03:26:25.544150 4108 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260810 03:26:25.554680 4108 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.161.20:37811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786332336319544-28292-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786332336319544-28292-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.161.62:41823
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.161.62:41823
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786332336319544-28292-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.161.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786332336319544-28292-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786332336319544-28292-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:26:25.556098 4108 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:26:25.557480 4108 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:26:25.569530 4119 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:26:25.569572 4120 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:26:25.569602 4123 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:26:25.570119 4108 server_base.cc:1034] running on GCE node
I20260810 03:26:25.570955 4108 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:26:25.572369 4108 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:26:25.573568 4108 hybrid_clock.cc:648] HybridClock initialized: now 1786332385573515 us; error 57 us; skew 500 ppm
I20260810 03:26:25.573908 4108 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 03:26:25.579375 4108 webserver.cc:460] Webserver started at http://127.27.161.62:42257/ using document root <none> and password file <none>
I20260810 03:26:25.580168 4108 fs_manager.cc:359] Metadata directory not provided
I20260810 03:26:25.580284 4108 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:26:25.580673 4108 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:26:25.583782 4108 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786332336319544-28292-0/minicluster-data/master-0/data/instance:
uuid: "fd8ea8ff70a84eff9bac995cea1f9198"
format_stamp: "Formatted at 2026-08-10 03:26:25 on dist-test-slave-vpvm"
server_key: "de51c9f88ced8e02f2d4250dbb9a3257"
server_key_iv: "d308fe19abcdcc064dd25223438ab163"
server_key_version: "encryptionkey@0"
I20260810 03:26:25.584609 4108 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786332336319544-28292-0/minicluster-data/master-0/wal/instance:
uuid: "fd8ea8ff70a84eff9bac995cea1f9198"
format_stamp: "Formatted at 2026-08-10 03:26:25 on dist-test-slave-vpvm"
server_key: "de51c9f88ced8e02f2d4250dbb9a3257"
server_key_iv: "d308fe19abcdcc064dd25223438ab163"
server_key_version: "encryptionkey@0"
I20260810 03:26:25.589936 4108 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260810 03:26:25.593495 4130 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:26:25.594975 4108 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260810 03:26:25.595130 4108 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786332336319544-28292-0/minicluster-data/master-0/data,/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786332336319544-28292-0/minicluster-data/master-0/wal
uuid: "fd8ea8ff70a84eff9bac995cea1f9198"
format_stamp: "Formatted at 2026-08-10 03:26:25 on dist-test-slave-vpvm"
server_key: "de51c9f88ced8e02f2d4250dbb9a3257"
server_key_iv: "d308fe19abcdcc064dd25223438ab163"
server_key_version: "encryptionkey@0"
I20260810 03:26:25.595322 4108 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786332336319544-28292-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786332336319544-28292-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786332336319544-28292-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:26:25.612869 4108 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:26:25.613757 4108 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:26:25.614042 4108 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:26:25.633132 4108 rpc_server.cc:307] RPC server started. Bound to: 127.27.161.62:41823
I20260810 03:26:25.633152 4212 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.161.62:41823 every 8 connection(s)
I20260810 03:26:25.634608 4108 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786332336319544-28292-0/minicluster-data/master-0/data/info.pb
I20260810 03:26:25.638414 4213 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:26:25.641703 28292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iBH5/build/asan/bin/kudu as pid 4108
I20260810 03:26:25.641901 28292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786332336319544-28292-0/minicluster-data/master-0/wal/instance
I20260810 03:26:25.642773 28292 external_mini_cluster.cc:1428] Setting key f47be3d2a6c7a428d8fe0f2791b0187d
I20260810 03:26:25.651289 4213 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd8ea8ff70a84eff9bac995cea1f9198: Bootstrap starting.
I20260810 03:26:25.654521 4213 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fd8ea8ff70a84eff9bac995cea1f9198: Neither blocks nor log segments found. Creating new log.
I20260810 03:26:25.655772 4213 log.cc:824] T 00000000000000000000000000000000 P fd8ea8ff70a84eff9bac995cea1f9198: Log is configured to *not* fsync() on all Append() calls
I20260810 03:26:25.656332 4217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:25.644976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36322 (local address 127.27.161.62:41823)
0810 03:26:25.646049 (+ 1073us) server_negotiation.cc:206] Beginning negotiation
0810 03:26:25.646061 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 03:26:25.646114 (+ 53us) server_negotiation.cc:407] Connection header received
0810 03:26:25.647801 (+ 1687us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:26:25.647826 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:26:25.648321 (+ 495us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:26:25.648729 (+ 408us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:26:25.649323 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:25.650645 (+ 1322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:25.651952 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:25.652611 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:25.652809 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:25.653888 (+ 1079us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:26:25.653895 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:26:25.653910 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:26:25.654498 (+ 588us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:26:25.654593 (+ 95us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:26:25.654615 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:26:25.654746 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0810 03:26:25.654971 (+ 225us) server_negotiation.cc:299] Negotiation successful
0810 03:26:25.655287 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":590,"thread_start_us":347,"threads_started":1}
I20260810 03:26:25.659049 4213 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd8ea8ff70a84eff9bac995cea1f9198: No bootstrap required, opened a new log
I20260810 03:26:25.660246 28292 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 03:26:25.667151 4213 raft_consensus.cc:348] T 00000000000000000000000000000000 P fd8ea8ff70a84eff9bac995cea1f9198 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd8ea8ff70a84eff9bac995cea1f9198" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 41823 } }
I20260810 03:26:25.667562 4213 raft_consensus.cc:374] T 00000000000000000000000000000000 P fd8ea8ff70a84eff9bac995cea1f9198 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:26:25.667660 4213 raft_consensus.cc:729] T 00000000000000000000000000000000 P fd8ea8ff70a84eff9bac995cea1f9198 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd8ea8ff70a84eff9bac995cea1f9198, State: Initialized, Role: FOLLOWER
I20260810 03:26:25.668257 4213 consensus_queue.cc:260] T 00000000000000000000000000000000 P fd8ea8ff70a84eff9bac995cea1f9198 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd8ea8ff70a84eff9bac995cea1f9198" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 41823 } }
I20260810 03:26:25.668483 4213 raft_consensus.cc:388] T 00000000000000000000000000000000 P fd8ea8ff70a84eff9bac995cea1f9198 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 03:26:25.668622 4213 raft_consensus.cc:482] T 00000000000000000000000000000000 P fd8ea8ff70a84eff9bac995cea1f9198 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 03:26:25.668780 4213 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fd8ea8ff70a84eff9bac995cea1f9198 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:26:25.671197 4213 raft_consensus.cc:504] T 00000000000000000000000000000000 P fd8ea8ff70a84eff9bac995cea1f9198 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd8ea8ff70a84eff9bac995cea1f9198" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 41823 } }
I20260810 03:26:25.671732 4213 leader_election.cc:302] T 00000000000000000000000000000000 P fd8ea8ff70a84eff9bac995cea1f9198 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fd8ea8ff70a84eff9bac995cea1f9198; no voters:
I20260810 03:26:25.672333 4213 leader_election.cc:288] T 00000000000000000000000000000000 P fd8ea8ff70a84eff9bac995cea1f9198 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 03:26:25.672468 4219 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fd8ea8ff70a84eff9bac995cea1f9198 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:26:25.673313 4219 raft_consensus.cc:686] T 00000000000000000000000000000000 P fd8ea8ff70a84eff9bac995cea1f9198 [term 1 LEADER]: Becoming Leader. State: Replica: fd8ea8ff70a84eff9bac995cea1f9198, State: Running, Role: LEADER
I20260810 03:26:25.673820 4219 consensus_queue.cc:237] T 00000000000000000000000000000000 P fd8ea8ff70a84eff9bac995cea1f9198 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd8ea8ff70a84eff9bac995cea1f9198" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 41823 } }
I20260810 03:26:25.674652 4213 sys_catalog.cc:564] T 00000000000000000000000000000000 P fd8ea8ff70a84eff9bac995cea1f9198 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:26:25.677205 4224 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd8ea8ff70a84eff9bac995cea1f9198 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fd8ea8ff70a84eff9bac995cea1f9198. Latest consensus state: current_term: 1 leader_uuid: "fd8ea8ff70a84eff9bac995cea1f9198" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd8ea8ff70a84eff9bac995cea1f9198" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 41823 } } }
I20260810 03:26:25.677284 4222 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd8ea8ff70a84eff9bac995cea1f9198 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fd8ea8ff70a84eff9bac995cea1f9198" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd8ea8ff70a84eff9bac995cea1f9198" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 41823 } } }
I20260810 03:26:25.677567 4224 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd8ea8ff70a84eff9bac995cea1f9198 [sys.catalog]: This master's current role is: LEADER
I20260810 03:26:25.677644 4222 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd8ea8ff70a84eff9bac995cea1f9198 [sys.catalog]: This master's current role is: LEADER
I20260810 03:26:25.679879 4237 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 03:26:25.685128 4237 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 03:26:25.694957 4237 catalog_manager.cc:1347] Generated new cluster ID: 7d7c1cfc3d954a3e9311ce7e41be5d97
I20260810 03:26:25.695101 4237 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 03:26:25.706533 4237 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 03:26:25.707916 4237 catalog_manager.cc:1504] Loading token signing keys...
I20260810 03:26:25.715853 4237 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fd8ea8ff70a84eff9bac995cea1f9198: Generated new TSK 0
I20260810 03:26:25.716789 4237 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 03:26:25.780297 4217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:25.735662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36336 (local address 127.27.161.62:41823)
0810 03:26:25.735862 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0810 03:26:25.735869 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:26:25.736249 (+ 380us) server_negotiation.cc:407] Connection header received
0810 03:26:25.736509 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:26:25.736514 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:26:25.736601 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:26:25.736746 (+ 145us) server_negotiation.cc:226] Negotiated authn=JWT
0810 03:26:25.737227 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:25.738034 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:25.739408 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:25.739854 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:25.739973 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:25.740500 (+ 527us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0810 03:26:25.779477 (+ 38977us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0810 03:26:25.779948 (+ 471us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:36336: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":59}
W20260810 03:26:25.780479 4217 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:36336: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260810 03:26:25.780577 4263 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:26:25.735535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:41823 (local address 127.0.0.1:36336)
0810 03:26:25.736133 (+ 598us) negotiation.cc:107] Waiting for socket to connect
0810 03:26:25.736150 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 03:26:25.736308 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:26:25.736781 (+ 473us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:26:25.736785 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:26:25.736799 (+ 14us) client_negotiation.cc:189] Negotiated authn=JWT
0810 03:26:25.737071 (+ 272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:26:25.737080 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:26:25.738181 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:26:25.738186 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:26:25.739251 (+ 1065us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:26:25.739263 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:26:25.740048 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:26:25.740054 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:26:25.740194 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:25.740232 (+ 38us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0810 03:26:25.779820 (+ 39588us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0810 03:26:25.780143 (+ 323us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.27.161.62:41823: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":316,"threads_started":1}
I20260810 03:26:25.781894 28292 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iBH5/build/asan/bin/kudu pid 4108
I20260810 03:26:25.783288 4206 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:26:25.880642 28292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iBH5/build/asan/bin/kudu with pid 4108
2026-08-10T03:26:25Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (778 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (443 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (437 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786332336319544-28292-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 03:26:26 dist-test-slave-vpvm krb5kdc[4289](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 03:26:26 dist-test-slave-vpvm krb5kdc[4289](info): set up 2 sockets
Aug 10 03:26:26 dist-test-slave-vpvm krb5kdc[4289](info): commencing operation
krb5kdc: starting...
W20260810 03:26:27.998705 28292 mini_kdc.cc:121] Time spent starting KDC: real 1.192s user 0.000s sys 0.039s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786332336319544-28292-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786332336319544-28292-0/krb5kdc/test-user.keytab.
Aug 10 03:26:28 dist-test-slave-vpvm krb5kdc[4289](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332388, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T03:26:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T03:26:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.161.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.161.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.161.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.62.keytab.
Entry for principal kudu/127.27.161.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.62.keytab.
I20260810 03:26:28.261564 28292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iBH5/build/asan/bin/kudu
/tmp/dist-test-task24iBH5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786332336319544-28292-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786332336319544-28292-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786332336319544-28292-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786332336319544-28292-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.161.62:36617
--webserver_interface=127.27.161.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.62.keytab
--principal=kudu/127.27.161.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.161.20:44705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.161.62:36617
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786332336319544-28292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786332336319544-28292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786332336319544-28292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:26:28.605868 4306 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:26:28.606261 4306 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:26:28.606406 4306 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:26:28.613881 4306 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:26:28.614043 4306 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:26:28.614128 4306 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:26:28.614207 4306 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:26:28.614266 4306 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:26:28.625000 4306 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.161.20:44705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786332336319544-28292-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786332336319544-28292-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.161.62:36617
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.62.keytab
--principal=kudu/127.27.161.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.161.62:36617
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786332336319544-28292-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.161.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786332336319544-28292-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:26:28.626585 4306 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:26:28.628038 4306 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:26:28.640403 4311 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:26:28.640424 4313 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:26:28.640674 4315 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:26:28.641670 4306 server_base.cc:1034] running on GCE node
I20260810 03:26:28.642526 4306 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:26:28.644048 4306 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:26:28.645262 4306 hybrid_clock.cc:648] HybridClock initialized: now 1786332388645212 us; error 43 us; skew 500 ppm
I20260810 03:26:28.645622 4306 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:26:28 dist-test-slave-vpvm krb5kdc[4289](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332388, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.161.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:26:28.648712 4306 init.cc:377] Logged in from keytab as kudu/127.27.161.62@KRBTEST.COM (short username kudu)
I20260810 03:26:28.650368 4306 webserver.cc:460] Webserver started at http://127.27.161.62:44069/ using document root <none> and password file <none>
I20260810 03:26:28.651039 4306 fs_manager.cc:359] Metadata directory not provided
I20260810 03:26:28.651149 4306 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:26:28.651427 4306 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:26:28.654575 4306 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786332336319544-28292-0/minicluster-data/master-0/data/instance:
uuid: "5abe24e73780494abfbf94957c9af4d6"
format_stamp: "Formatted at 2026-08-10 03:26:28 on dist-test-slave-vpvm"
server_key: "bf4e922bfa18e4fb6b6c1bf507b46e18"
server_key_iv: "00ebda4d7afd33a6408ce539dcc0c945"
server_key_version: "encryptionkey@0"
I20260810 03:26:28.655351 4306 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786332336319544-28292-0/minicluster-data/master-0/wal/instance:
uuid: "5abe24e73780494abfbf94957c9af4d6"
format_stamp: "Formatted at 2026-08-10 03:26:28 on dist-test-slave-vpvm"
server_key: "bf4e922bfa18e4fb6b6c1bf507b46e18"
server_key_iv: "00ebda4d7afd33a6408ce539dcc0c945"
server_key_version: "encryptionkey@0"
I20260810 03:26:28.661115 4306 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.002s
I20260810 03:26:28.664785 4335 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:26:28.666404 4306 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 03:26:28.666606 4306 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786332336319544-28292-0/minicluster-data/master-0/data,/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786332336319544-28292-0/minicluster-data/master-0/wal
uuid: "5abe24e73780494abfbf94957c9af4d6"
format_stamp: "Formatted at 2026-08-10 03:26:28 on dist-test-slave-vpvm"
server_key: "bf4e922bfa18e4fb6b6c1bf507b46e18"
server_key_iv: "00ebda4d7afd33a6408ce539dcc0c945"
server_key_version: "encryptionkey@0"
I20260810 03:26:28.666823 4306 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786332336319544-28292-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786332336319544-28292-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786332336319544-28292-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:26:28.703121 4306 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:26:28.707814 4306 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:26:28.708109 4306 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:26:28.726740 4306 rpc_server.cc:307] RPC server started. Bound to: 127.27.161.62:36617
I20260810 03:26:28.726770 4417 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.161.62:36617 every 8 connection(s)
I20260810 03:26:28.728243 4306 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786332336319544-28292-0/minicluster-data/master-0/data/info.pb
I20260810 03:26:28.728930 28292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iBH5/build/asan/bin/kudu as pid 4306
I20260810 03:26:28.729138 28292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786332336319544-28292-0/minicluster-data/master-0/wal/instance
I20260810 03:26:28.729872 28292 external_mini_cluster.cc:1428] Setting key 9564b801d032ced1414631df2d9e4432
I20260810 03:26:28.732807 4418 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 10 03:26:28 dist-test-slave-vpvm krb5kdc[4289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332388, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.161.62@KRBTEST.COM
I20260810 03:26:28.746439 4418 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5abe24e73780494abfbf94957c9af4d6: Bootstrap starting.
I20260810 03:26:28.750801 4418 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5abe24e73780494abfbf94957c9af4d6: Neither blocks nor log segments found. Creating new log.
I20260810 03:26:28.751982 4418 log.cc:824] T 00000000000000000000000000000000 P 5abe24e73780494abfbf94957c9af4d6: Log is configured to *not* fsync() on all Append() calls
I20260810 03:26:28.755613 4421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:28.731979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37782 (local address 127.27.161.62:36617)
0810 03:26:28.732809 (+ 830us) server_negotiation.cc:206] Beginning negotiation
0810 03:26:28.732824 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 03:26:28.732878 (+ 54us) server_negotiation.cc:407] Connection header received
0810 03:26:28.735270 (+ 2392us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:26:28.735302 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:26:28.735828 (+ 526us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:26:28.736298 (+ 470us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:26:28.737704 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:28.739772 (+ 2068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:28.741009 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:28.742058 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:28.742241 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:28.746423 (+ 4182us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:26:28.746436 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:26:28.746458 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:26:28.746520 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:26:28.750458 (+ 3938us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:26:28.751315 (+ 857us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:26:28.751325 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:26:28.751333 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:26:28.751548 (+ 215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:26:28.752064 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:26:28.752069 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:26:28.752072 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:26:28.752863 (+ 791us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:26:28.753120 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0810 03:26:28.753710 (+ 590us) server_negotiation.cc:299] Negotiation successful
0810 03:26:28.754211 (+ 501us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":577,"thread_start_us":331,"threads_started":1}
I20260810 03:26:28.755802 4418 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5abe24e73780494abfbf94957c9af4d6: No bootstrap required, opened a new log
I20260810 03:26:28.764300 4418 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5abe24e73780494abfbf94957c9af4d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5abe24e73780494abfbf94957c9af4d6" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 36617 } }
I20260810 03:26:28.764725 4418 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5abe24e73780494abfbf94957c9af4d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:26:28.764830 4418 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5abe24e73780494abfbf94957c9af4d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5abe24e73780494abfbf94957c9af4d6, State: Initialized, Role: FOLLOWER
I20260810 03:26:28.765519 4418 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5abe24e73780494abfbf94957c9af4d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5abe24e73780494abfbf94957c9af4d6" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 36617 } }
I20260810 03:26:28.765793 4418 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5abe24e73780494abfbf94957c9af4d6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 03:26:28.765966 4418 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5abe24e73780494abfbf94957c9af4d6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 03:26:28.766158 4418 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5abe24e73780494abfbf94957c9af4d6 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:26:28.768594 4418 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5abe24e73780494abfbf94957c9af4d6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5abe24e73780494abfbf94957c9af4d6" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 36617 } }
I20260810 03:26:28.769200 4418 leader_election.cc:302] T 00000000000000000000000000000000 P 5abe24e73780494abfbf94957c9af4d6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5abe24e73780494abfbf94957c9af4d6; no voters:
I20260810 03:26:28.769917 4418 leader_election.cc:288] T 00000000000000000000000000000000 P 5abe24e73780494abfbf94957c9af4d6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 03:26:28.770100 4423 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5abe24e73780494abfbf94957c9af4d6 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:26:28.770905 4423 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5abe24e73780494abfbf94957c9af4d6 [term 1 LEADER]: Becoming Leader. State: Replica: 5abe24e73780494abfbf94957c9af4d6, State: Running, Role: LEADER
I20260810 03:26:28.771507 4423 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5abe24e73780494abfbf94957c9af4d6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5abe24e73780494abfbf94957c9af4d6" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 36617 } }
I20260810 03:26:28.772322 4418 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5abe24e73780494abfbf94957c9af4d6 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:26:28.774664 4426 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5abe24e73780494abfbf94957c9af4d6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5abe24e73780494abfbf94957c9af4d6. Latest consensus state: current_term: 1 leader_uuid: "5abe24e73780494abfbf94957c9af4d6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5abe24e73780494abfbf94957c9af4d6" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 36617 } } }
I20260810 03:26:28.774744 4425 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5abe24e73780494abfbf94957c9af4d6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5abe24e73780494abfbf94957c9af4d6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5abe24e73780494abfbf94957c9af4d6" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 36617 } } }
I20260810 03:26:28.775110 4426 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5abe24e73780494abfbf94957c9af4d6 [sys.catalog]: This master's current role is: LEADER
I20260810 03:26:28.775207 4425 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5abe24e73780494abfbf94957c9af4d6 [sys.catalog]: This master's current role is: LEADER
I20260810 03:26:28.776794 4431 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 03:26:28.782922 4431 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 03:26:28.783743 4450 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260810 03:26:28.792681 4431 catalog_manager.cc:1347] Generated new cluster ID: 2b2e8504dbee488db6155c1afc55aa2b
I20260810 03:26:28.792805 4431 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 03:26:28.809244 4431 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 03:26:28.810999 4431 catalog_manager.cc:1504] Loading token signing keys...
I20260810 03:26:28.822707 4431 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5abe24e73780494abfbf94957c9af4d6: Generated new TSK 0
I20260810 03:26:28.823822 4431 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.161.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.161.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.161.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.1.keytab.
Entry for principal kudu/127.27.161.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.1.keytab.
I20260810 03:26:28.895983 28292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iBH5/build/asan/bin/kudu
/tmp/dist-test-task24iBH5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786332336319544-28292-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786332336319544-28292-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786332336319544-28292-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786332336319544-28292-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.161.1:0
--local_ip_for_outbound_sockets=127.27.161.1
--webserver_interface=127.27.161.1
--webserver_port=0
--tserver_master_addrs=127.27.161.62:36617
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.1.keytab
--principal=kudu/127.27.161.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.161.20:44705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786332336319544-28292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786332336319544-28292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786332336319544-28292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:26:29.257155 4459 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:26:29.257603 4459 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:26:29.257736 4459 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:26:29.265455 4459 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:26:29.265586 4459 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:26:29.265714 4459 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.161.1
I20260810 03:26:29.276232 4459 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.161.20:44705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786332336319544-28292-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786332336319544-28292-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.1.keytab
--principal=kudu/127.27.161.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.161.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786332336319544-28292-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.161.1
--webserver_port=0
--tserver_master_addrs=127.27.161.62:36617
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.161.1
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786332336319544-28292-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:26:29.277572 4459 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:26:29.279040 4459 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:26:29.292919 4465 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:26:29.292951 4468 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:26:29.292886 4466 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:26:29.293823 4459 server_base.cc:1034] running on GCE node
I20260810 03:26:29.294505 4459 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:26:29.295527 4459 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:26:29.296731 4459 hybrid_clock.cc:648] HybridClock initialized: now 1786332389296711 us; error 84 us; skew 500 ppm
I20260810 03:26:29.297056 4459 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:26:29 dist-test-slave-vpvm krb5kdc[4289](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332389, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.161.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:26:29.300213 4459 init.cc:377] Logged in from keytab as kudu/127.27.161.1@KRBTEST.COM (short username kudu)
I20260810 03:26:29.301810 4459 webserver.cc:460] Webserver started at http://127.27.161.1:46389/ using document root <none> and password file <none>
I20260810 03:26:29.302541 4459 fs_manager.cc:359] Metadata directory not provided
I20260810 03:26:29.302649 4459 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:26:29.302914 4459 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:26:29.305998 4459 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786332336319544-28292-0/minicluster-data/ts-0/data/instance:
uuid: "bd024b93f17d4fb4a645b0862ec2ea31"
format_stamp: "Formatted at 2026-08-10 03:26:29 on dist-test-slave-vpvm"
server_key: "489133b812fdec9b79fc941f91bebf23"
server_key_iv: "509329b85d3fddccc6c733c1ce21bd27"
server_key_version: "encryptionkey@0"
I20260810 03:26:29.306820 4459 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786332336319544-28292-0/minicluster-data/ts-0/wal/instance:
uuid: "bd024b93f17d4fb4a645b0862ec2ea31"
format_stamp: "Formatted at 2026-08-10 03:26:29 on dist-test-slave-vpvm"
server_key: "489133b812fdec9b79fc941f91bebf23"
server_key_iv: "509329b85d3fddccc6c733c1ce21bd27"
server_key_version: "encryptionkey@0"
I20260810 03:26:29.312493 4459 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260810 03:26:29.316519 4482 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:26:29.318166 4459 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260810 03:26:29.318346 4459 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786332336319544-28292-0/minicluster-data/ts-0/data,/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786332336319544-28292-0/minicluster-data/ts-0/wal
uuid: "bd024b93f17d4fb4a645b0862ec2ea31"
format_stamp: "Formatted at 2026-08-10 03:26:29 on dist-test-slave-vpvm"
server_key: "489133b812fdec9b79fc941f91bebf23"
server_key_iv: "509329b85d3fddccc6c733c1ce21bd27"
server_key_version: "encryptionkey@0"
I20260810 03:26:29.318545 4459 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786332336319544-28292-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786332336319544-28292-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786332336319544-28292-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:26:29.335516 4459 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:26:29.339869 4459 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:26:29.340142 4459 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:26:29.341156 4459 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:26:29.342548 4459 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:26:29.342625 4459 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:26:29.342724 4459 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:26:29.342772 4459 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:26:29.375193 4459 rpc_server.cc:307] RPC server started. Bound to: 127.27.161.1:40481
I20260810 03:26:29.375231 4637 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.161.1:40481 every 8 connection(s)
I20260810 03:26:29.376526 4459 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786332336319544-28292-0/minicluster-data/ts-0/data/info.pb
I20260810 03:26:29.386112 28292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iBH5/build/asan/bin/kudu as pid 4459
I20260810 03:26:29.386345 28292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786332336319544-28292-0/minicluster-data/ts-0/wal/instance
I20260810 03:26:29.387228 28292 external_mini_cluster.cc:1428] Setting key 62bb199238d7c6b153d6be35bb949509
Aug 10 03:26:29 dist-test-slave-vpvm krb5kdc[4289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332389, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.161.1@KRBTEST.COM for kudu/127.27.161.62@KRBTEST.COM
I20260810 03:26:29.397426 4643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:29.379168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.161.1:40789 (local address 127.27.161.62:36617)
0810 03:26:29.379670 (+ 502us) server_negotiation.cc:206] Beginning negotiation
0810 03:26:29.379677 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:26:29.380707 (+ 1030us) server_negotiation.cc:407] Connection header received
0810 03:26:29.382575 (+ 1868us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:26:29.382580 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:26:29.382658 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:26:29.382759 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:26:29.384392 (+ 1633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:29.385121 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:29.386895 (+ 1774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:29.387323 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:29.387550 (+ 227us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:29.391551 (+ 4001us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:26:29.391556 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:26:29.391559 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:26:29.391594 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:26:29.394208 (+ 2614us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:26:29.395006 (+ 798us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:26:29.395011 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:26:29.395013 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:26:29.395147 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:26:29.395608 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:26:29.395612 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:26:29.395614 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:26:29.395902 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:26:29.396051 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0810 03:26:29.396808 (+ 757us) server_negotiation.cc:299] Negotiation successful
0810 03:26:29.397069 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":227,"threads_started":1}
I20260810 03:26:29.398501 4644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:29.379724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:36617 (local address 127.27.161.1:40789)
0810 03:26:29.380512 (+ 788us) negotiation.cc:107] Waiting for socket to connect
0810 03:26:29.380565 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0810 03:26:29.382307 (+ 1742us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:26:29.383029 (+ 722us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:26:29.383038 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:26:29.383621 (+ 583us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:26:29.384237 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:26:29.384256 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:26:29.385302 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:26:29.385307 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:26:29.386715 (+ 1408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:26:29.386729 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:26:29.387670 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:26:29.387678 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:26:29.387965 (+ 287us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:29.388681 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:26:29.388712 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:26:29.391304 (+ 2592us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:26:29.394376 (+ 3072us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:26:29.394387 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:26:29.394402 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:26:29.394867 (+ 465us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:26:29.395260 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:26:29.395266 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:26:29.395270 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:26:29.395482 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:26:29.396034 (+ 552us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:26:29.396043 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:26:29.396404 (+ 361us) client_negotiation.cc:769] Sending connection context
0810 03:26:29.396663 (+ 259us) client_negotiation.cc:240] Negotiation successful
0810 03:26:29.397252 (+ 589us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":513,"thread_start_us":306,"threads_started":1}
I20260810 03:26:29.400142 4641 heartbeater.cc:347] Connected to a master server at 127.27.161.62:36617
I20260810 03:26:29.400496 4641 heartbeater.cc:464] Registering TS with master...
I20260810 03:26:29.401412 4641 heartbeater.cc:511] Master 127.27.161.62:36617 requested a full tablet report, sending...
I20260810 03:26:29.403923 4365 ts_manager.cc:194] Registered new tserver with Master: bd024b93f17d4fb4a645b0862ec2ea31 (127.27.161.1:40481)
I20260810 03:26:29.405524 4365 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.161.1@KRBTEST.COM'} at 127.27.161.1:40789
WARNING: no policy specified for kudu/127.27.161.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.161.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.161.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.2.keytab.
Entry for principal kudu/127.27.161.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.2.keytab.
I20260810 03:26:29.445494 28292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iBH5/build/asan/bin/kudu
/tmp/dist-test-task24iBH5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786332336319544-28292-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786332336319544-28292-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786332336319544-28292-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786332336319544-28292-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.161.2:0
--local_ip_for_outbound_sockets=127.27.161.2
--webserver_interface=127.27.161.2
--webserver_port=0
--tserver_master_addrs=127.27.161.62:36617
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.2.keytab
--principal=kudu/127.27.161.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.161.20:44705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786332336319544-28292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786332336319544-28292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786332336319544-28292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:26:29.791419 4648 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:26:29.791723 4648 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:26:29.791838 4648 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:26:29.799263 4648 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:26:29.799393 4648 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:26:29.799551 4648 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.161.2
I20260810 03:26:29.809818 4648 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.161.20:44705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786332336319544-28292-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786332336319544-28292-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.2.keytab
--principal=kudu/127.27.161.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.161.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786332336319544-28292-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.161.2
--webserver_port=0
--tserver_master_addrs=127.27.161.62:36617
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.161.2
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786332336319544-28292-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:26:29.811259 4648 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:26:29.812637 4648 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:26:29.826344 4659 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:26:29.826411 4660 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:26:29.826617 4662 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:26:29.827349 4648 server_base.cc:1034] running on GCE node
I20260810 03:26:29.827970 4648 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:26:29.828914 4648 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:26:29.830116 4648 hybrid_clock.cc:648] HybridClock initialized: now 1786332389830058 us; error 48 us; skew 500 ppm
I20260810 03:26:29.830448 4648 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:26:29 dist-test-slave-vpvm krb5kdc[4289](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332389, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.161.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:26:29.833587 4648 init.cc:377] Logged in from keytab as kudu/127.27.161.2@KRBTEST.COM (short username kudu)
I20260810 03:26:29.835184 4648 webserver.cc:460] Webserver started at http://127.27.161.2:46797/ using document root <none> and password file <none>
I20260810 03:26:29.835912 4648 fs_manager.cc:359] Metadata directory not provided
I20260810 03:26:29.836016 4648 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:26:29.836289 4648 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:26:29.839296 4648 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786332336319544-28292-0/minicluster-data/ts-1/data/instance:
uuid: "8d29b5ab83be47e5914d34866d6a78b8"
format_stamp: "Formatted at 2026-08-10 03:26:29 on dist-test-slave-vpvm"
server_key: "919ad973b6b56938074293f883ce3462"
server_key_iv: "213e07bcd77b74d47aaef57a3b9a9135"
server_key_version: "encryptionkey@0"
I20260810 03:26:29.840111 4648 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786332336319544-28292-0/minicluster-data/ts-1/wal/instance:
uuid: "8d29b5ab83be47e5914d34866d6a78b8"
format_stamp: "Formatted at 2026-08-10 03:26:29 on dist-test-slave-vpvm"
server_key: "919ad973b6b56938074293f883ce3462"
server_key_iv: "213e07bcd77b74d47aaef57a3b9a9135"
server_key_version: "encryptionkey@0"
I20260810 03:26:29.845368 4648 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260810 03:26:29.849009 4671 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:26:29.850535 4648 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 03:26:29.850710 4648 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786332336319544-28292-0/minicluster-data/ts-1/data,/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786332336319544-28292-0/minicluster-data/ts-1/wal
uuid: "8d29b5ab83be47e5914d34866d6a78b8"
format_stamp: "Formatted at 2026-08-10 03:26:29 on dist-test-slave-vpvm"
server_key: "919ad973b6b56938074293f883ce3462"
server_key_iv: "213e07bcd77b74d47aaef57a3b9a9135"
server_key_version: "encryptionkey@0"
I20260810 03:26:29.850906 4648 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786332336319544-28292-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786332336319544-28292-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786332336319544-28292-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:26:29.865285 4648 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:26:29.869747 4648 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:26:29.870059 4648 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:26:29.871176 4648 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:26:29.872722 4648 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:26:29.872824 4648 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:26:29.872936 4648 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:26:29.872984 4648 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:26:29.905372 4648 rpc_server.cc:307] RPC server started. Bound to: 127.27.161.2:38263
I20260810 03:26:29.905406 4836 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.161.2:38263 every 8 connection(s)
I20260810 03:26:29.906718 4648 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786332336319544-28292-0/minicluster-data/ts-1/data/info.pb
I20260810 03:26:29.913420 28292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iBH5/build/asan/bin/kudu as pid 4648
I20260810 03:26:29.913609 28292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786332336319544-28292-0/minicluster-data/ts-1/wal/instance
I20260810 03:26:29.914191 28292 external_mini_cluster.cc:1428] Setting key bbb0f3599c9f43122d68b9d2a9e41e48
Aug 10 03:26:29 dist-test-slave-vpvm krb5kdc[4289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332389, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.161.2@KRBTEST.COM for kudu/127.27.161.62@KRBTEST.COM
I20260810 03:26:29.928490 4841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:29.909324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.161.2:38421 (local address 127.27.161.62:36617)
0810 03:26:29.909809 (+ 485us) server_negotiation.cc:206] Beginning negotiation
0810 03:26:29.909813 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:26:29.910753 (+ 940us) server_negotiation.cc:407] Connection header received
0810 03:26:29.912571 (+ 1818us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:26:29.912576 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:26:29.912659 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:26:29.912757 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:26:29.914859 (+ 2102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:29.915876 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:29.918156 (+ 2280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:29.918600 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:29.918715 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:29.922770 (+ 4055us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:26:29.922775 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:26:29.922779 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:26:29.922814 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:26:29.925303 (+ 2489us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:26:29.926071 (+ 768us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:26:29.926076 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:26:29.926078 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:26:29.926222 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:26:29.926675 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:26:29.926679 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:26:29.926680 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:26:29.926948 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:26:29.927085 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0810 03:26:29.927833 (+ 748us) server_negotiation.cc:299] Negotiation successful
0810 03:26:29.928079 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":368,"thread_start_us":247,"threads_started":1}
I20260810 03:26:29.929487 4843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:29.909814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:36617 (local address 127.27.161.2:38421)
0810 03:26:29.910587 (+ 773us) negotiation.cc:107] Waiting for socket to connect
0810 03:26:29.910633 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0810 03:26:29.912302 (+ 1669us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:26:29.913028 (+ 726us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:26:29.913040 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:26:29.913794 (+ 754us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:26:29.914653 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:26:29.914682 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:26:29.916063 (+ 1381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:26:29.916069 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:26:29.918002 (+ 1933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:26:29.918021 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:26:29.918855 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:26:29.918863 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:26:29.919159 (+ 296us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:29.919905 (+ 746us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:26:29.919933 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:26:29.922506 (+ 2573us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:26:29.925454 (+ 2948us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:26:29.925464 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:26:29.925483 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:26:29.925922 (+ 439us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:26:29.926340 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:26:29.926345 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:26:29.926349 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:26:29.926554 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:26:29.927076 (+ 522us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:26:29.927084 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:26:29.927454 (+ 370us) client_negotiation.cc:769] Sending connection context
0810 03:26:29.927710 (+ 256us) client_negotiation.cc:240] Negotiation successful
0810 03:26:29.928276 (+ 566us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":526,"thread_start_us":323,"threads_started":1}
I20260810 03:26:29.930995 4837 heartbeater.cc:347] Connected to a master server at 127.27.161.62:36617
I20260810 03:26:29.931339 4837 heartbeater.cc:464] Registering TS with master...
I20260810 03:26:29.932416 4837 heartbeater.cc:511] Master 127.27.161.62:36617 requested a full tablet report, sending...
I20260810 03:26:29.934026 4365 ts_manager.cc:194] Registered new tserver with Master: 8d29b5ab83be47e5914d34866d6a78b8 (127.27.161.2:38263)
I20260810 03:26:29.934847 4365 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.161.2@KRBTEST.COM'} at 127.27.161.2:38421
WARNING: no policy specified for kudu/127.27.161.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.161.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.161.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.3.keytab.
Entry for principal kudu/127.27.161.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.3.keytab.
I20260810 03:26:29.973626 28292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iBH5/build/asan/bin/kudu
/tmp/dist-test-task24iBH5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786332336319544-28292-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786332336319544-28292-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786332336319544-28292-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786332336319544-28292-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.161.3:0
--local_ip_for_outbound_sockets=127.27.161.3
--webserver_interface=127.27.161.3
--webserver_port=0
--tserver_master_addrs=127.27.161.62:36617
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.3.keytab
--principal=kudu/127.27.161.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.161.20:44705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786332336319544-28292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786332336319544-28292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786332336319544-28292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:26:30.335212 4846 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:26:30.335546 4846 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:26:30.335656 4846 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:26:30.343252 4846 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:26:30.343405 4846 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:26:30.343593 4846 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.161.3
I20260810 03:26:30.354235 4846 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.161.20:44705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786332336319544-28292-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786332336319544-28292-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.3.keytab
--principal=kudu/127.27.161.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.161.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786332336319544-28292-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.161.3
--webserver_port=0
--tserver_master_addrs=127.27.161.62:36617
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.161.3
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786332336319544-28292-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:26:30.355685 4846 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:26:30.357091 4846 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:26:30.370731 4854 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:26:30.370668 4856 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:26:30.370865 4858 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:26:30.371757 4846 server_base.cc:1034] running on GCE node
I20260810 03:26:30.372432 4846 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:26:30.373471 4846 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:26:30.374728 4846 hybrid_clock.cc:648] HybridClock initialized: now 1786332390374651 us; error 42 us; skew 500 ppm
I20260810 03:26:30.375082 4846 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:26:30 dist-test-slave-vpvm krb5kdc[4289](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332390, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.161.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:26:30.378333 4846 init.cc:377] Logged in from keytab as kudu/127.27.161.3@KRBTEST.COM (short username kudu)
I20260810 03:26:30.379926 4846 webserver.cc:460] Webserver started at http://127.27.161.3:44895/ using document root <none> and password file <none>
I20260810 03:26:30.380648 4846 fs_manager.cc:359] Metadata directory not provided
I20260810 03:26:30.380795 4846 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:26:30.381570 4846 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:26:30.385345 4846 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786332336319544-28292-0/minicluster-data/ts-2/data/instance:
uuid: "2954d536a7b24a10a71b7f967718c7f4"
format_stamp: "Formatted at 2026-08-10 03:26:30 on dist-test-slave-vpvm"
server_key: "7dc90522555410c436556e5509595df8"
server_key_iv: "cc2018160cc70c52013219e30551bd2d"
server_key_version: "encryptionkey@0"
I20260810 03:26:30.386281 4846 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786332336319544-28292-0/minicluster-data/ts-2/wal/instance:
uuid: "2954d536a7b24a10a71b7f967718c7f4"
format_stamp: "Formatted at 2026-08-10 03:26:30 on dist-test-slave-vpvm"
server_key: "7dc90522555410c436556e5509595df8"
server_key_iv: "cc2018160cc70c52013219e30551bd2d"
server_key_version: "encryptionkey@0"
I20260810 03:26:30.392077 4846 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260810 03:26:30.395903 4866 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:26:30.397295 4846 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260810 03:26:30.397456 4846 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786332336319544-28292-0/minicluster-data/ts-2/data,/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786332336319544-28292-0/minicluster-data/ts-2/wal
uuid: "2954d536a7b24a10a71b7f967718c7f4"
format_stamp: "Formatted at 2026-08-10 03:26:30 on dist-test-slave-vpvm"
server_key: "7dc90522555410c436556e5509595df8"
server_key_iv: "cc2018160cc70c52013219e30551bd2d"
server_key_version: "encryptionkey@0"
I20260810 03:26:30.397675 4846 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786332336319544-28292-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786332336319544-28292-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786332336319544-28292-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:26:30.408906 4641 heartbeater.cc:503] Master 127.27.161.62:36617 was elected leader, sending a full tablet report...
I20260810 03:26:30.438304 4846 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:26:30.442668 4846 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:26:30.442925 4846 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:26:30.443899 4846 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:26:30.445217 4846 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:26:30.445290 4846 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:26:30.445381 4846 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:26:30.445425 4846 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:26:30.478744 4846 rpc_server.cc:307] RPC server started. Bound to: 127.27.161.3:39455
I20260810 03:26:30.478812 5046 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.161.3:39455 every 8 connection(s)
I20260810 03:26:30.479990 4846 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786332336319544-28292-0/minicluster-data/ts-2/data/info.pb
I20260810 03:26:30.482861 28292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iBH5/build/asan/bin/kudu as pid 4846
I20260810 03:26:30.483057 28292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786332336319544-28292-0/minicluster-data/ts-2/wal/instance
I20260810 03:26:30.483659 28292 external_mini_cluster.cc:1428] Setting key 57e32f087f7e3aee1c7f447f237377d2
Aug 10 03:26:30 dist-test-slave-vpvm krb5kdc[4289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332390, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.161.3@KRBTEST.COM for kudu/127.27.161.62@KRBTEST.COM
I20260810 03:26:30.500725 5051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:30.482534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.161.3:59661 (local address 127.27.161.62:36617)
0810 03:26:30.483054 (+ 520us) server_negotiation.cc:206] Beginning negotiation
0810 03:26:30.483062 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 03:26:30.484130 (+ 1068us) server_negotiation.cc:407] Connection header received
0810 03:26:30.486382 (+ 2252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:26:30.486388 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:26:30.486513 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:26:30.486637 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:26:30.488220 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:30.489080 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:30.490653 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:30.491068 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:30.491175 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:30.494671 (+ 3496us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:26:30.494677 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:26:30.494680 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:26:30.494718 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:26:30.497403 (+ 2685us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:26:30.498372 (+ 969us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:26:30.498376 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:26:30.498379 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:26:30.498511 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:26:30.499039 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:26:30.499044 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:26:30.499047 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:26:30.499438 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:26:30.499564 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0810 03:26:30.500262 (+ 698us) server_negotiation.cc:299] Negotiation successful
0810 03:26:30.500440 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":376,"thread_start_us":253,"threads_started":1}
I20260810 03:26:30.501806 5053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:30.483100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:36617 (local address 127.27.161.3:59661)
0810 03:26:30.483962 (+ 862us) negotiation.cc:107] Waiting for socket to connect
0810 03:26:30.484008 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0810 03:26:30.486091 (+ 2083us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:26:30.486859 (+ 768us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:26:30.486872 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:26:30.487428 (+ 556us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:26:30.488006 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:26:30.488023 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:26:30.489226 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:26:30.489230 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:26:30.490500 (+ 1270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:26:30.490513 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:26:30.491276 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:26:30.491281 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:26:30.491501 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:30.492123 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:26:30.492147 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:26:30.494403 (+ 2256us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:26:30.497579 (+ 3176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:26:30.497589 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:26:30.497608 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:26:30.498250 (+ 642us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:26:30.498642 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:26:30.498646 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:26:30.498650 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:26:30.498914 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:26:30.499567 (+ 653us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:26:30.499574 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:26:30.499914 (+ 340us) client_negotiation.cc:769] Sending connection context
0810 03:26:30.500172 (+ 258us) client_negotiation.cc:240] Negotiation successful
0810 03:26:30.500625 (+ 453us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":592,"thread_start_us":351,"threads_started":1}
I20260810 03:26:30.503144 5047 heartbeater.cc:347] Connected to a master server at 127.27.161.62:36617
I20260810 03:26:30.503481 5047 heartbeater.cc:464] Registering TS with master...
I20260810 03:26:30.504323 5047 heartbeater.cc:511] Master 127.27.161.62:36617 requested a full tablet report, sending...
I20260810 03:26:30.505909 4365 ts_manager.cc:194] Registered new tserver with Master: 2954d536a7b24a10a71b7f967718c7f4 (127.27.161.3:39455)
I20260810 03:26:30.506790 4365 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.161.3@KRBTEST.COM'} at 127.27.161.3:59661
I20260810 03:26:30.511600 28292 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786332336319544-28292-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786332336319544-28292-0/krb5kdc/test-admin.keytab.
Aug 10 03:26:30 dist-test-slave-vpvm krb5kdc[4289](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332390, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:26:30.543417 28292 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260810 03:26:30.549304 5060 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:26:30.546346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40748 (local address 127.27.161.1:40481)
0810 03:26:30.547040 (+ 694us) server_negotiation.cc:206] Beginning negotiation
0810 03:26:30.547051 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 03:26:30.547081 (+ 30us) server_negotiation.cc:407] Connection header received
0810 03:26:30.547150 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:26:30.547158 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:26:30.547368 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:26:30.547538 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:26:30.548789 (+ 1251us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40748: BlockingRecv error: recv got EOF from 127.0.0.1:40748 (error 108)
Metrics: {"server-negotiator.queue_time_us":490,"thread_start_us":293,"threads_started":1}
W20260810 03:26:30.549398 28292 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786332336319544-28292-0/krb5kdc/krb5cc' of type FILE
Aug 10 03:26:30 dist-test-slave-vpvm krb5kdc[4289](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332390, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:26:30.551242 28292 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 10 03:26:30 dist-test-slave-vpvm krb5kdc[4289](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332390, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:26:30.552762 28292 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 10 03:26:30 dist-test-slave-vpvm krb5kdc[4289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332390, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.161.1@KRBTEST.COM
I20260810 03:26:30.570461 5060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:30.554797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40758 (local address 127.27.161.1:40481)
0810 03:26:30.555035 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0810 03:26:30.555042 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:26:30.555141 (+ 99us) server_negotiation.cc:407] Connection header received
0810 03:26:30.555369 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:26:30.555375 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:26:30.555463 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:26:30.555569 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:26:30.556742 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:30.557778 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:30.559164 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:30.559797 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:30.559929 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:30.563860 (+ 3931us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:26:30.563877 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:26:30.563894 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:26:30.563959 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:26:30.567304 (+ 3345us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:26:30.568107 (+ 803us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:26:30.568116 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:26:30.568119 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:26:30.568268 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:26:30.568675 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:26:30.568680 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:26:30.568685 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:26:30.569119 (+ 434us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:26:30.569425 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0810 03:26:30.569798 (+ 373us) server_negotiation.cc:299] Negotiation successful
0810 03:26:30.570139 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
W20260810 03:26:30.576860 5060 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:26:30.574970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40770 (local address 127.27.161.1:40481)
0810 03:26:30.575195 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0810 03:26:30.575199 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:26:30.575312 (+ 113us) server_negotiation.cc:407] Connection header received
0810 03:26:30.575527 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:26:30.575532 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:26:30.575627 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:26:30.575716 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:26:30.576613 (+ 897us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40770: BlockingRecv error: recv got EOF from 127.0.0.1:40770 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
W20260810 03:26:30.577006 28292 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786332336319544-28292-0/krb5kdc/krb5cc' of type FILE
Aug 10 03:26:30 dist-test-slave-vpvm krb5kdc[4289](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332390, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:26:30.578765 28292 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 10 03:26:30 dist-test-slave-vpvm krb5kdc[4289](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332390, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:26:30.580251 28292 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 10 03:26:30 dist-test-slave-vpvm krb5kdc[4289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332390, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.161.1@KRBTEST.COM
I20260810 03:26:30.598282 5060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:30.582074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40782 (local address 127.27.161.1:40481)
0810 03:26:30.582291 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0810 03:26:30.582297 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:26:30.582430 (+ 133us) server_negotiation.cc:407] Connection header received
0810 03:26:30.582629 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:26:30.582635 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:26:30.582704 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:26:30.582788 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:26:30.583792 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:30.584542 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:30.585783 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:30.586284 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:30.586400 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:30.592908 (+ 6508us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:26:30.592913 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:26:30.592917 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:26:30.592977 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:26:30.595611 (+ 2634us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:26:30.596324 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:26:30.596330 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:26:30.596334 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:26:30.596472 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:26:30.596870 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:26:30.596875 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:26:30.596878 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:26:30.597185 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:26:30.597449 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0810 03:26:30.597740 (+ 291us) server_negotiation.cc:299] Negotiation successful
0810 03:26:30.597997 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
W20260810 03:26:30.602973 5060 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:26:30.601032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40790 (local address 127.27.161.1:40481)
0810 03:26:30.601181 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0810 03:26:30.601185 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:26:30.601565 (+ 380us) server_negotiation.cc:407] Connection header received
0810 03:26:30.601735 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:26:30.601739 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:26:30.601844 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:26:30.601934 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:26:30.602754 (+ 820us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40790: BlockingRecv error: recv got EOF from 127.0.0.1:40790 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260810 03:26:30.603191 28292 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786332336319544-28292-0/krb5kdc/krb5cc' of type FILE
Aug 10 03:26:30 dist-test-slave-vpvm krb5kdc[4289](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332390, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:26:30.604966 28292 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 10 03:26:30 dist-test-slave-vpvm krb5kdc[4289](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332390, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:26:30.606405 28292 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 10 03:26:30 dist-test-slave-vpvm krb5kdc[4289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332390, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.161.1@KRBTEST.COM
I20260810 03:26:30.622764 5060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:30.607845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40800 (local address 127.27.161.1:40481)
0810 03:26:30.608006 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0810 03:26:30.608010 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:26:30.608360 (+ 350us) server_negotiation.cc:407] Connection header received
0810 03:26:30.608546 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:26:30.608552 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:26:30.608631 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:26:30.608725 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:26:30.609771 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:30.610624 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:30.611996 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:30.612444 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:30.612736 (+ 292us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:30.616617 (+ 3881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:26:30.616623 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:26:30.616627 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:26:30.616676 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:26:30.619578 (+ 2902us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:26:30.620390 (+ 812us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:26:30.620395 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:26:30.620397 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:26:30.620596 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:26:30.621039 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:26:30.621044 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:26:30.621046 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:26:30.621403 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:26:30.621819 (+ 416us) server_negotiation.cc:1035] Waiting for connection context
0810 03:26:30.622142 (+ 323us) server_negotiation.cc:299] Negotiation successful
0810 03:26:30.622397 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260810 03:26:30.624729 28292 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iBH5/build/asan/bin/kudu pid 4459
I20260810 03:26:30.637538 5060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:30.625640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40808 (local address 127.27.161.1:40481)
0810 03:26:30.625793 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0810 03:26:30.625797 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:26:30.625999 (+ 202us) server_negotiation.cc:407] Connection header received
0810 03:26:30.626228 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:26:30.626232 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:26:30.626300 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:26:30.626389 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:26:30.627507 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:30.628248 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:30.629545 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:30.630091 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:30.630207 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:30.632320 (+ 2113us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:26:30.632325 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:26:30.632329 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:26:30.632366 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:26:30.634931 (+ 2565us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:26:30.635702 (+ 771us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:26:30.635707 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:26:30.635710 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:26:30.635863 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:26:30.636234 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:26:30.636239 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:26:30.636241 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:26:30.636565 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:26:30.636790 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0810 03:26:30.637087 (+ 297us) server_negotiation.cc:299] Negotiation successful
0810 03:26:30.637271 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260810 03:26:30.638213 4631 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:26:30.727166 28292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iBH5/build/asan/bin/kudu with pid 4459
I20260810 03:26:30.753937 28292 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iBH5/build/asan/bin/kudu pid 4648
Aug 10 03:26:30 dist-test-slave-vpvm krb5kdc[4289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332390, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.161.2@KRBTEST.COM
I20260810 03:26:30.769321 5083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:30.754968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50282 (local address 127.27.161.2:38263)
0810 03:26:30.755526 (+ 558us) server_negotiation.cc:206] Beginning negotiation
0810 03:26:30.755536 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 03:26:30.755566 (+ 30us) server_negotiation.cc:407] Connection header received
0810 03:26:30.755634 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:26:30.755640 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:26:30.755845 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:26:30.755980 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:26:30.756933 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:30.757937 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:30.759236 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:30.759807 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:30.759904 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:30.763800 (+ 3896us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:26:30.763815 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:26:30.763831 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:26:30.763869 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:26:30.766645 (+ 2776us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:26:30.767305 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:26:30.767312 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:26:30.767314 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:26:30.767438 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:26:30.767877 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:26:30.767882 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:26:30.767884 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:26:30.768298 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:26:30.768566 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0810 03:26:30.768793 (+ 227us) server_negotiation.cc:299] Negotiation successful
0810 03:26:30.769006 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":401,"thread_start_us":283,"threads_started":1}
I20260810 03:26:30.770798 4829 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:26:30.854557 28292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iBH5/build/asan/bin/kudu with pid 4648
I20260810 03:26:30.876556 28292 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iBH5/build/asan/bin/kudu pid 4846
Aug 10 03:26:30 dist-test-slave-vpvm krb5kdc[4289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332390, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.161.3@KRBTEST.COM
I20260810 03:26:30.892144 5087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:30.877530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53818 (local address 127.27.161.3:39455)
0810 03:26:30.878250 (+ 720us) server_negotiation.cc:206] Beginning negotiation
0810 03:26:30.878258 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 03:26:30.878288 (+ 30us) server_negotiation.cc:407] Connection header received
0810 03:26:30.878351 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:26:30.878357 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:26:30.878571 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:26:30.878709 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:26:30.879844 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:30.880866 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:30.882259 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:30.882837 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:30.882939 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:30.886807 (+ 3868us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:26:30.886820 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:26:30.886837 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:26:30.886874 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:26:30.889476 (+ 2602us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:26:30.890244 (+ 768us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:26:30.890252 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:26:30.890255 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:26:30.890397 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:26:30.890729 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:26:30.890732 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:26:30.890734 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:26:30.891109 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:26:30.891363 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0810 03:26:30.891532 (+ 169us) server_negotiation.cc:299] Negotiation successful
0810 03:26:30.891793 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":559,"thread_start_us":356,"threads_started":1}
I20260810 03:26:30.893427 5040 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:26:30.976117 28292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iBH5/build/asan/bin/kudu with pid 4846
I20260810 03:26:30.999922 28292 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iBH5/build/asan/bin/kudu pid 4306
I20260810 03:26:31.001206 4411 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:26:31.106526 28292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iBH5/build/asan/bin/kudu with pid 4306
2026-08-10T03:26:31Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (4350 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786332336319544-28292-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 03:26:31 dist-test-slave-vpvm krb5kdc[5096](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 03:26:31 dist-test-slave-vpvm krb5kdc[5096](info): set up 2 sockets
Aug 10 03:26:31 dist-test-slave-vpvm krb5kdc[5096](info): commencing operation
krb5kdc: starting...
W20260810 03:26:32.333654 28292 mini_kdc.cc:121] Time spent starting KDC: real 1.177s user 0.000s sys 0.037s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786332336319544-28292-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786332336319544-28292-0/krb5kdc/test-user.keytab.
Aug 10 03:26:32 dist-test-slave-vpvm krb5kdc[5096](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332392, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T03:26:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T03:26:32Z Disabled control of system clock
WARNING: no policy specified for oryx/127.27.161.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.161.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.161.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786332336319544-28292-0/krb5kdc/oryx_127.27.161.62.keytab.
Entry for principal oryx/127.27.161.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786332336319544-28292-0/krb5kdc/oryx_127.27.161.62.keytab.
I20260810 03:26:32.611493 28292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iBH5/build/asan/bin/kudu
/tmp/dist-test-task24iBH5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786332336319544-28292-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786332336319544-28292-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786332336319544-28292-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786332336319544-28292-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.161.62:36281
--webserver_interface=127.27.161.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786332336319544-28292-0/krb5kdc/oryx_127.27.161.62.keytab
--principal=oryx/127.27.161.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.161.20:43095
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.161.62:36281
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786332336319544-28292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786332336319544-28292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786332336319544-28292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:26:32.955228 5113 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:26:32.955557 5113 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:26:32.955691 5113 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:26:32.962977 5113 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:26:32.963099 5113 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:26:32.963178 5113 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:26:32.963282 5113 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:26:32.963341 5113 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:26:32.973228 5113 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.161.20:43095
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786332336319544-28292-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786332336319544-28292-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.161.62:36281
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786332336319544-28292-0/krb5kdc/oryx_127.27.161.62.keytab
--principal=oryx/127.27.161.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.161.62:36281
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786332336319544-28292-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.161.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786332336319544-28292-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:26:32.974612 5113 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:26:32.975984 5113 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:26:32.987511 5120 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:26:32.987509 5122 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:26:32.987813 5124 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:26:32.988234 5113 server_base.cc:1034] running on GCE node
I20260810 03:26:32.989003 5113 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:26:32.990514 5113 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:26:32.991741 5113 hybrid_clock.cc:648] HybridClock initialized: now 1786332392991681 us; error 46 us; skew 500 ppm
I20260810 03:26:32.992077 5113 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:26:32 dist-test-slave-vpvm krb5kdc[5096](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332392, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.161.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:26:32.995339 5113 init.cc:377] Logged in from keytab as oryx/127.27.161.62@KRBTEST.COM (short username oryx)
I20260810 03:26:32.996980 5113 webserver.cc:460] Webserver started at http://127.27.161.62:44977/ using document root <none> and password file <none>
I20260810 03:26:32.997771 5113 fs_manager.cc:359] Metadata directory not provided
I20260810 03:26:32.997896 5113 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:26:32.998322 5113 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:26:33.001493 5113 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786332336319544-28292-0/minicluster-data/master-0/data/instance:
uuid: "4b65187fc20346eda207dd58ee639ef2"
format_stamp: "Formatted at 2026-08-10 03:26:32 on dist-test-slave-vpvm"
server_key: "fc0d5b0de0d95f98cae6b054fc80cea3"
server_key_iv: "72e750c7b2db79cfceadb0534c9aa5e9"
server_key_version: "encryptionkey@0"
I20260810 03:26:33.002346 5113 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786332336319544-28292-0/minicluster-data/master-0/wal/instance:
uuid: "4b65187fc20346eda207dd58ee639ef2"
format_stamp: "Formatted at 2026-08-10 03:26:32 on dist-test-slave-vpvm"
server_key: "fc0d5b0de0d95f98cae6b054fc80cea3"
server_key_iv: "72e750c7b2db79cfceadb0534c9aa5e9"
server_key_version: "encryptionkey@0"
I20260810 03:26:33.008279 5113 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260810 03:26:33.012503 5133 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:26:33.014079 5113 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 03:26:33.014287 5113 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786332336319544-28292-0/minicluster-data/master-0/data,/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786332336319544-28292-0/minicluster-data/master-0/wal
uuid: "4b65187fc20346eda207dd58ee639ef2"
format_stamp: "Formatted at 2026-08-10 03:26:32 on dist-test-slave-vpvm"
server_key: "fc0d5b0de0d95f98cae6b054fc80cea3"
server_key_iv: "72e750c7b2db79cfceadb0534c9aa5e9"
server_key_version: "encryptionkey@0"
I20260810 03:26:33.014494 5113 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786332336319544-28292-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786332336319544-28292-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786332336319544-28292-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:26:33.056531 5113 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:26:33.060710 5113 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:26:33.060981 5113 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:26:33.079941 5113 rpc_server.cc:307] RPC server started. Bound to: 127.27.161.62:36281
I20260810 03:26:33.079967 5209 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.161.62:36281 every 8 connection(s)
I20260810 03:26:33.081378 5113 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786332336319544-28292-0/minicluster-data/master-0/data/info.pb
I20260810 03:26:33.085300 5211 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:26:33.089630 28292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iBH5/build/asan/bin/kudu as pid 5113
I20260810 03:26:33.089855 28292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786332336319544-28292-0/minicluster-data/master-0/wal/instance
I20260810 03:26:33.090732 28292 external_mini_cluster.cc:1428] Setting key d6277127caf375b2e0cc9a7ed6aae489
I20260810 03:26:33.097548 5211 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b65187fc20346eda207dd58ee639ef2: Bootstrap starting.
I20260810 03:26:33.101382 5211 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4b65187fc20346eda207dd58ee639ef2: Neither blocks nor log segments found. Creating new log.
I20260810 03:26:33.102653 5211 log.cc:824] T 00000000000000000000000000000000 P 4b65187fc20346eda207dd58ee639ef2: Log is configured to *not* fsync() on all Append() calls
I20260810 03:26:33.106364 5211 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b65187fc20346eda207dd58ee639ef2: No bootstrap required, opened a new log
Aug 10 03:26:33 dist-test-slave-vpvm krb5kdc[5096](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332392, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.27.161.62@KRBTEST.COM
I20260810 03:26:33.116797 5216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:33.092952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54740 (local address 127.27.161.62:36281)
0810 03:26:33.093757 (+ 805us) server_negotiation.cc:206] Beginning negotiation
0810 03:26:33.093769 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 03:26:33.093815 (+ 46us) server_negotiation.cc:407] Connection header received
0810 03:26:33.095902 (+ 2087us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:26:33.095928 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:26:33.096349 (+ 421us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:26:33.096680 (+ 331us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:26:33.098608 (+ 1928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:33.099949 (+ 1341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:33.101816 (+ 1867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:33.102637 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:33.102820 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:33.107524 (+ 4704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:26:33.107540 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:26:33.107560 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:26:33.107606 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:26:33.111553 (+ 3947us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:26:33.112280 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:26:33.112290 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:26:33.112299 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:26:33.112525 (+ 226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:26:33.112880 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:26:33.112885 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:26:33.112889 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:26:33.113694 (+ 805us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:26:33.114056 (+ 362us) server_negotiation.cc:1035] Waiting for connection context
0810 03:26:33.114506 (+ 450us) server_negotiation.cc:299] Negotiation successful
0810 03:26:33.115295 (+ 789us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":593,"thread_start_us":371,"threads_started":1}
I20260810 03:26:33.117354 5211 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4b65187fc20346eda207dd58ee639ef2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b65187fc20346eda207dd58ee639ef2" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 36281 } }
I20260810 03:26:33.117863 5211 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4b65187fc20346eda207dd58ee639ef2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:26:33.118069 5211 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4b65187fc20346eda207dd58ee639ef2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b65187fc20346eda207dd58ee639ef2, State: Initialized, Role: FOLLOWER
I20260810 03:26:33.118837 5211 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4b65187fc20346eda207dd58ee639ef2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b65187fc20346eda207dd58ee639ef2" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 36281 } }
I20260810 03:26:33.119086 5211 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4b65187fc20346eda207dd58ee639ef2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 03:26:33.119272 5211 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4b65187fc20346eda207dd58ee639ef2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 03:26:33.119452 5211 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4b65187fc20346eda207dd58ee639ef2 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:26:33.121820 5211 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4b65187fc20346eda207dd58ee639ef2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b65187fc20346eda207dd58ee639ef2" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 36281 } }
I20260810 03:26:33.122396 5211 leader_election.cc:302] T 00000000000000000000000000000000 P 4b65187fc20346eda207dd58ee639ef2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4b65187fc20346eda207dd58ee639ef2; no voters:
I20260810 03:26:33.122991 5211 leader_election.cc:288] T 00000000000000000000000000000000 P 4b65187fc20346eda207dd58ee639ef2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 03:26:33.123142 5219 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4b65187fc20346eda207dd58ee639ef2 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:26:33.124619 5219 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4b65187fc20346eda207dd58ee639ef2 [term 1 LEADER]: Becoming Leader. State: Replica: 4b65187fc20346eda207dd58ee639ef2, State: Running, Role: LEADER
I20260810 03:26:33.125298 5211 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4b65187fc20346eda207dd58ee639ef2 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:26:33.125411 5219 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4b65187fc20346eda207dd58ee639ef2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b65187fc20346eda207dd58ee639ef2" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 36281 } }
I20260810 03:26:33.128248 5221 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b65187fc20346eda207dd58ee639ef2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4b65187fc20346eda207dd58ee639ef2. Latest consensus state: current_term: 1 leader_uuid: "4b65187fc20346eda207dd58ee639ef2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b65187fc20346eda207dd58ee639ef2" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 36281 } } }
I20260810 03:26:33.128664 5221 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b65187fc20346eda207dd58ee639ef2 [sys.catalog]: This master's current role is: LEADER
I20260810 03:26:33.129279 5220 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b65187fc20346eda207dd58ee639ef2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4b65187fc20346eda207dd58ee639ef2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b65187fc20346eda207dd58ee639ef2" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 36281 } } }
I20260810 03:26:33.129630 5220 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b65187fc20346eda207dd58ee639ef2 [sys.catalog]: This master's current role is: LEADER
I20260810 03:26:33.134346 5234 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 03:26:33.140199 5234 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 03:26:33.149714 5234 catalog_manager.cc:1347] Generated new cluster ID: 75ff020955394fb1943d772cebcad889
I20260810 03:26:33.149853 5234 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 03:26:33.179220 5234 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 03:26:33.181304 5234 catalog_manager.cc:1504] Loading token signing keys...
I20260810 03:26:33.192030 5234 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4b65187fc20346eda207dd58ee639ef2: Generated new TSK 0
I20260810 03:26:33.193148 5234 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.27.161.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.161.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.161.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786332336319544-28292-0/krb5kdc/oryx_127.27.161.1.keytab.
Entry for principal oryx/127.27.161.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786332336319544-28292-0/krb5kdc/oryx_127.27.161.1.keytab.
I20260810 03:26:33.258515 28292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iBH5/build/asan/bin/kudu
/tmp/dist-test-task24iBH5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786332336319544-28292-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786332336319544-28292-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786332336319544-28292-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786332336319544-28292-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.161.1:0
--local_ip_for_outbound_sockets=127.27.161.1
--webserver_interface=127.27.161.1
--webserver_port=0
--tserver_master_addrs=127.27.161.62:36281
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786332336319544-28292-0/krb5kdc/oryx_127.27.161.1.keytab
--principal=oryx/127.27.161.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.161.20:43095
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786332336319544-28292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786332336319544-28292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786332336319544-28292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:26:33.613742 5247 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:26:33.614149 5247 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:26:33.614307 5247 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:26:33.622165 5247 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:26:33.622288 5247 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:26:33.622442 5247 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.161.1
I20260810 03:26:33.633313 5247 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.161.20:43095
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786332336319544-28292-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786332336319544-28292-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786332336319544-28292-0/krb5kdc/oryx_127.27.161.1.keytab
--principal=oryx/127.27.161.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.161.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786332336319544-28292-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.161.1
--webserver_port=0
--tserver_master_addrs=127.27.161.62:36281
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.161.1
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786332336319544-28292-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:26:33.634850 5247 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:26:33.636392 5247 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:26:33.650214 5258 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:26:33.650287 5255 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:26:33.650260 5253 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:26:33.651063 5247 server_base.cc:1034] running on GCE node
I20260810 03:26:33.651741 5247 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:26:33.652815 5247 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:26:33.654040 5247 hybrid_clock.cc:648] HybridClock initialized: now 1786332393653987 us; error 43 us; skew 500 ppm
I20260810 03:26:33.654381 5247 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:26:33 dist-test-slave-vpvm krb5kdc[5096](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332393, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.161.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:26:33.660493 5247 init.cc:377] Logged in from keytab as oryx/127.27.161.1@KRBTEST.COM (short username oryx)
I20260810 03:26:33.662277 5247 webserver.cc:460] Webserver started at http://127.27.161.1:39437/ using document root <none> and password file <none>
I20260810 03:26:33.663148 5247 fs_manager.cc:359] Metadata directory not provided
I20260810 03:26:33.663311 5247 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:26:33.663637 5247 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:26:33.666891 5247 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786332336319544-28292-0/minicluster-data/ts-0/data/instance:
uuid: "b7227f1049a6433489fae09b53501439"
format_stamp: "Formatted at 2026-08-10 03:26:33 on dist-test-slave-vpvm"
server_key: "c35f815820600d7c401954e211e9b3fb"
server_key_iv: "13fd7bbc96ac0a033eb66632ebd97c9f"
server_key_version: "encryptionkey@0"
I20260810 03:26:33.667730 5247 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786332336319544-28292-0/minicluster-data/ts-0/wal/instance:
uuid: "b7227f1049a6433489fae09b53501439"
format_stamp: "Formatted at 2026-08-10 03:26:33 on dist-test-slave-vpvm"
server_key: "c35f815820600d7c401954e211e9b3fb"
server_key_iv: "13fd7bbc96ac0a033eb66632ebd97c9f"
server_key_version: "encryptionkey@0"
I20260810 03:26:33.673563 5247 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260810 03:26:33.677608 5270 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:26:33.679255 5247 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260810 03:26:33.679461 5247 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786332336319544-28292-0/minicluster-data/ts-0/data,/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786332336319544-28292-0/minicluster-data/ts-0/wal
uuid: "b7227f1049a6433489fae09b53501439"
format_stamp: "Formatted at 2026-08-10 03:26:33 on dist-test-slave-vpvm"
server_key: "c35f815820600d7c401954e211e9b3fb"
server_key_iv: "13fd7bbc96ac0a033eb66632ebd97c9f"
server_key_version: "encryptionkey@0"
I20260810 03:26:33.679711 5247 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786332336319544-28292-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786332336319544-28292-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786332336319544-28292-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:26:33.692811 5247 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:26:33.697727 5247 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:26:33.698104 5247 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:26:33.699177 5247 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:26:33.700742 5247 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:26:33.700825 5247 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:26:33.700959 5247 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:26:33.701020 5247 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:26:33.734799 5247 rpc_server.cc:307] RPC server started. Bound to: 127.27.161.1:36623
I20260810 03:26:33.734872 5443 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.161.1:36623 every 8 connection(s)
I20260810 03:26:33.736143 5247 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786332336319544-28292-0/minicluster-data/ts-0/data/info.pb
I20260810 03:26:33.736752 28292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iBH5/build/asan/bin/kudu as pid 5247
I20260810 03:26:33.736943 28292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786332336319544-28292-0/minicluster-data/ts-0/wal/instance
I20260810 03:26:33.737854 28292 external_mini_cluster.cc:1428] Setting key e975ab720a4a27566a337ec83bc399d1
Aug 10 03:26:33 dist-test-slave-vpvm krb5kdc[5096](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332393, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.161.1@KRBTEST.COM for oryx/127.27.161.62@KRBTEST.COM
I20260810 03:26:33.759768 5449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:33.738954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.161.1:45631 (local address 127.27.161.62:36281)
0810 03:26:33.739454 (+ 500us) server_negotiation.cc:206] Beginning negotiation
0810 03:26:33.739461 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:26:33.740726 (+ 1265us) server_negotiation.cc:407] Connection header received
0810 03:26:33.742778 (+ 2052us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:26:33.742786 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:26:33.742887 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:26:33.743043 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:26:33.744977 (+ 1934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:33.746048 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:33.747755 (+ 1707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:33.748418 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:33.748580 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:33.752882 (+ 4302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:26:33.752888 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:26:33.752892 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:26:33.752933 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:26:33.755770 (+ 2837us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:26:33.756929 (+ 1159us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:26:33.756935 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:26:33.756937 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:26:33.757089 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:26:33.757697 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:26:33.757713 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:26:33.757715 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:26:33.758129 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:26:33.758268 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0810 03:26:33.759254 (+ 986us) server_negotiation.cc:299] Negotiation successful
0810 03:26:33.759472 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":364,"thread_start_us":243,"threads_started":1}
I20260810 03:26:33.761274 5450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:33.739620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:36281 (local address 127.27.161.1:45631)
0810 03:26:33.740532 (+ 912us) negotiation.cc:107] Waiting for socket to connect
0810 03:26:33.740584 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0810 03:26:33.742426 (+ 1842us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:26:33.743495 (+ 1069us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:26:33.743505 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:26:33.744081 (+ 576us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:26:33.744733 (+ 652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:26:33.744754 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:26:33.746245 (+ 1491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:26:33.746249 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:26:33.747566 (+ 1317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:26:33.747581 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:26:33.748656 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:26:33.748662 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:26:33.748878 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:33.749611 (+ 733us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:26:33.749637 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:26:33.752533 (+ 2896us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:26:33.755973 (+ 3440us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:26:33.755986 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:26:33.756007 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:26:33.756772 (+ 765us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:26:33.757224 (+ 452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:26:33.757229 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:26:33.757232 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:26:33.757532 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:26:33.758278 (+ 746us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:26:33.758287 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:26:33.758801 (+ 514us) client_negotiation.cc:769] Sending connection context
0810 03:26:33.759128 (+ 327us) client_negotiation.cc:240] Negotiation successful
0810 03:26:33.759750 (+ 622us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":620,"thread_start_us":368,"threads_started":1}
I20260810 03:26:33.763271 5445 heartbeater.cc:347] Connected to a master server at 127.27.161.62:36281
I20260810 03:26:33.763674 5445 heartbeater.cc:464] Registering TS with master...
I20260810 03:26:33.764751 5445 heartbeater.cc:511] Master 127.27.161.62:36281 requested a full tablet report, sending...
WARNING: no policy specified for oryx/127.27.161.2@KRBTEST.COM; defaulting to no policy
I20260810 03:26:33.768086 5158 ts_manager.cc:194] Registered new tserver with Master: b7227f1049a6433489fae09b53501439 (127.27.161.1:36623)
I20260810 03:26:33.769713 5158 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.27.161.1@KRBTEST.COM'} at 127.27.161.1:45631
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.161.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.161.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786332336319544-28292-0/krb5kdc/oryx_127.27.161.2.keytab.
Entry for principal oryx/127.27.161.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786332336319544-28292-0/krb5kdc/oryx_127.27.161.2.keytab.
I20260810 03:26:33.805155 28292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iBH5/build/asan/bin/kudu
/tmp/dist-test-task24iBH5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786332336319544-28292-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786332336319544-28292-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786332336319544-28292-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786332336319544-28292-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.161.2:0
--local_ip_for_outbound_sockets=127.27.161.2
--webserver_interface=127.27.161.2
--webserver_port=0
--tserver_master_addrs=127.27.161.62:36281
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786332336319544-28292-0/krb5kdc/oryx_127.27.161.2.keytab
--principal=oryx/127.27.161.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.161.20:43095
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786332336319544-28292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786332336319544-28292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786332336319544-28292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:26:34.166888 5454 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:26:34.167183 5454 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:26:34.167294 5454 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:26:34.174661 5454 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:26:34.174780 5454 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:26:34.174902 5454 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.161.2
I20260810 03:26:34.185159 5454 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.161.20:43095
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786332336319544-28292-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786332336319544-28292-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786332336319544-28292-0/krb5kdc/oryx_127.27.161.2.keytab
--principal=oryx/127.27.161.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.161.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786332336319544-28292-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.161.2
--webserver_port=0
--tserver_master_addrs=127.27.161.62:36281
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.161.2
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786332336319544-28292-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:26:34.186647 5454 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:26:34.187968 5454 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:26:34.201244 5460 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:26:34.201200 5462 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:26:34.201277 5464 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:26:34.202107 5454 server_base.cc:1034] running on GCE node
I20260810 03:26:34.202718 5454 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:26:34.203761 5454 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:26:34.204963 5454 hybrid_clock.cc:648] HybridClock initialized: now 1786332394204912 us; error 49 us; skew 500 ppm
I20260810 03:26:34.205293 5454 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:26:34 dist-test-slave-vpvm krb5kdc[5096](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332394, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.161.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:26:34.208739 5454 init.cc:377] Logged in from keytab as oryx/127.27.161.2@KRBTEST.COM (short username oryx)
I20260810 03:26:34.210398 5454 webserver.cc:460] Webserver started at http://127.27.161.2:33517/ using document root <none> and password file <none>
I20260810 03:26:34.211179 5454 fs_manager.cc:359] Metadata directory not provided
I20260810 03:26:34.211306 5454 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:26:34.211673 5454 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:26:34.214905 5454 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786332336319544-28292-0/minicluster-data/ts-1/data/instance:
uuid: "b851ca7ac5ca41af917e9e02fb356ce8"
format_stamp: "Formatted at 2026-08-10 03:26:34 on dist-test-slave-vpvm"
server_key: "d2a274c144128afaa4de0942893dd2b8"
server_key_iv: "bad13effa24215efa14c62d2f917a0fd"
server_key_version: "encryptionkey@0"
I20260810 03:26:34.215682 5454 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786332336319544-28292-0/minicluster-data/ts-1/wal/instance:
uuid: "b851ca7ac5ca41af917e9e02fb356ce8"
format_stamp: "Formatted at 2026-08-10 03:26:34 on dist-test-slave-vpvm"
server_key: "d2a274c144128afaa4de0942893dd2b8"
server_key_iv: "bad13effa24215efa14c62d2f917a0fd"
server_key_version: "encryptionkey@0"
I20260810 03:26:34.221066 5454 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260810 03:26:34.224866 5473 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:26:34.226310 5454 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260810 03:26:34.226467 5454 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786332336319544-28292-0/minicluster-data/ts-1/data,/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786332336319544-28292-0/minicluster-data/ts-1/wal
uuid: "b851ca7ac5ca41af917e9e02fb356ce8"
format_stamp: "Formatted at 2026-08-10 03:26:34 on dist-test-slave-vpvm"
server_key: "d2a274c144128afaa4de0942893dd2b8"
server_key_iv: "bad13effa24215efa14c62d2f917a0fd"
server_key_version: "encryptionkey@0"
I20260810 03:26:34.226703 5454 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786332336319544-28292-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786332336319544-28292-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786332336319544-28292-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:26:34.238026 5454 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:26:34.241827 5454 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:26:34.242112 5454 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:26:34.243058 5454 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:26:34.244494 5454 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:26:34.244572 5454 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:26:34.244696 5454 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:26:34.244750 5454 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:26:34.287344 5454 rpc_server.cc:307] RPC server started. Bound to: 127.27.161.2:43345
I20260810 03:26:34.287380 5652 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.161.2:43345 every 8 connection(s)
I20260810 03:26:34.288661 5454 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786332336319544-28292-0/minicluster-data/ts-1/data/info.pb
I20260810 03:26:34.295300 28292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iBH5/build/asan/bin/kudu as pid 5454
I20260810 03:26:34.295480 28292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786332336319544-28292-0/minicluster-data/ts-1/wal/instance
I20260810 03:26:34.295943 28292 external_mini_cluster.cc:1428] Setting key f8885eeb6e38a0d08ef42368a317f892
Aug 10 03:26:34 dist-test-slave-vpvm krb5kdc[5096](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332394, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.161.2@KRBTEST.COM for oryx/127.27.161.62@KRBTEST.COM
I20260810 03:26:34.308331 5655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:34.291244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.161.2:38539 (local address 127.27.161.62:36281)
0810 03:26:34.291688 (+ 444us) server_negotiation.cc:206] Beginning negotiation
0810 03:26:34.291693 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:26:34.292545 (+ 852us) server_negotiation.cc:407] Connection header received
0810 03:26:34.294422 (+ 1877us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:26:34.294428 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:26:34.294513 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:26:34.294618 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:26:34.296275 (+ 1657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:34.297001 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:34.298629 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:34.299045 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:34.299161 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:34.302627 (+ 3466us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:26:34.302632 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:26:34.302635 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:26:34.302669 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:26:34.305160 (+ 2491us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:26:34.305887 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:26:34.305891 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:26:34.305893 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:26:34.306065 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:26:34.306498 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:26:34.306503 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:26:34.306506 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:26:34.306799 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:26:34.306937 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0810 03:26:34.307757 (+ 820us) server_negotiation.cc:299] Negotiation successful
0810 03:26:34.308001 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":338,"thread_start_us":225,"threads_started":1}
I20260810 03:26:34.309382 5657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:34.291686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:36281 (local address 127.27.161.2:38539)
0810 03:26:34.292385 (+ 699us) negotiation.cc:107] Waiting for socket to connect
0810 03:26:34.292432 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0810 03:26:34.294143 (+ 1711us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:26:34.294872 (+ 729us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:26:34.294880 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:26:34.295444 (+ 564us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:26:34.296101 (+ 657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:26:34.296122 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:26:34.297166 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:26:34.297170 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:26:34.298470 (+ 1300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:26:34.298489 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:26:34.299277 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:26:34.299285 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:26:34.299509 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:34.300149 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:26:34.300174 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:26:34.302387 (+ 2213us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:26:34.305299 (+ 2912us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:26:34.305309 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:26:34.305332 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:26:34.305756 (+ 424us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:26:34.306175 (+ 419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:26:34.306180 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:26:34.306184 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:26:34.306377 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:26:34.306933 (+ 556us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:26:34.306941 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:26:34.307318 (+ 377us) client_negotiation.cc:769] Sending connection context
0810 03:26:34.307593 (+ 275us) client_negotiation.cc:240] Negotiation successful
0810 03:26:34.308107 (+ 514us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":259,"threads_started":1}
I20260810 03:26:34.310971 5653 heartbeater.cc:347] Connected to a master server at 127.27.161.62:36281
I20260810 03:26:34.311293 5653 heartbeater.cc:464] Registering TS with master...
I20260810 03:26:34.312108 5653 heartbeater.cc:511] Master 127.27.161.62:36281 requested a full tablet report, sending...
I20260810 03:26:34.313730 5158 ts_manager.cc:194] Registered new tserver with Master: b851ca7ac5ca41af917e9e02fb356ce8 (127.27.161.2:43345)
I20260810 03:26:34.314628 5158 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.27.161.2@KRBTEST.COM'} at 127.27.161.2:38539
WARNING: no policy specified for oryx/127.27.161.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.161.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.161.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786332336319544-28292-0/krb5kdc/oryx_127.27.161.3.keytab.
Entry for principal oryx/127.27.161.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786332336319544-28292-0/krb5kdc/oryx_127.27.161.3.keytab.
I20260810 03:26:34.353296 28292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iBH5/build/asan/bin/kudu
/tmp/dist-test-task24iBH5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786332336319544-28292-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786332336319544-28292-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786332336319544-28292-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786332336319544-28292-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.161.3:0
--local_ip_for_outbound_sockets=127.27.161.3
--webserver_interface=127.27.161.3
--webserver_port=0
--tserver_master_addrs=127.27.161.62:36281
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786332336319544-28292-0/krb5kdc/oryx_127.27.161.3.keytab
--principal=oryx/127.27.161.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.161.20:43095
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786332336319544-28292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786332336319544-28292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786332336319544-28292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:26:34.706079 5663 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:26:34.706403 5663 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:26:34.706548 5663 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:26:34.714339 5663 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:26:34.714483 5663 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:26:34.714658 5663 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.161.3
I20260810 03:26:34.725157 5663 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.161.20:43095
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786332336319544-28292-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786332336319544-28292-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786332336319544-28292-0/krb5kdc/oryx_127.27.161.3.keytab
--principal=oryx/127.27.161.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.161.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786332336319544-28292-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.161.3
--webserver_port=0
--tserver_master_addrs=127.27.161.62:36281
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.161.3
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786332336319544-28292-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:26:34.726629 5663 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:26:34.728092 5663 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:26:34.742180 5668 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:26:34.742218 5672 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:26:34.742247 5669 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:26:34.742985 5663 server_base.cc:1034] running on GCE node
I20260810 03:26:34.743672 5663 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:26:34.744679 5663 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:26:34.745916 5663 hybrid_clock.cc:648] HybridClock initialized: now 1786332394745853 us; error 52 us; skew 500 ppm
I20260810 03:26:34.746302 5663 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:26:34 dist-test-slave-vpvm krb5kdc[5096](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332394, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.161.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:26:34.749392 5663 init.cc:377] Logged in from keytab as oryx/127.27.161.3@KRBTEST.COM (short username oryx)
I20260810 03:26:34.750910 5663 webserver.cc:460] Webserver started at http://127.27.161.3:40551/ using document root <none> and password file <none>
I20260810 03:26:34.751592 5663 fs_manager.cc:359] Metadata directory not provided
I20260810 03:26:34.751698 5663 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:26:34.751960 5663 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:26:34.755009 5663 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786332336319544-28292-0/minicluster-data/ts-2/data/instance:
uuid: "05f07b3f00144d75b6c6fc9a40632f17"
format_stamp: "Formatted at 2026-08-10 03:26:34 on dist-test-slave-vpvm"
server_key: "c71623682b543b69a57bb03ef3329687"
server_key_iv: "a8d2d31b59e20e9d16f868ed791d7ac0"
server_key_version: "encryptionkey@0"
I20260810 03:26:34.755765 5663 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786332336319544-28292-0/minicluster-data/ts-2/wal/instance:
uuid: "05f07b3f00144d75b6c6fc9a40632f17"
format_stamp: "Formatted at 2026-08-10 03:26:34 on dist-test-slave-vpvm"
server_key: "c71623682b543b69a57bb03ef3329687"
server_key_iv: "a8d2d31b59e20e9d16f868ed791d7ac0"
server_key_version: "encryptionkey@0"
I20260810 03:26:34.761076 5663 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260810 03:26:34.764700 5683 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:26:34.766065 5663 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 03:26:34.766215 5663 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786332336319544-28292-0/minicluster-data/ts-2/data,/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786332336319544-28292-0/minicluster-data/ts-2/wal
uuid: "05f07b3f00144d75b6c6fc9a40632f17"
format_stamp: "Formatted at 2026-08-10 03:26:34 on dist-test-slave-vpvm"
server_key: "c71623682b543b69a57bb03ef3329687"
server_key_iv: "a8d2d31b59e20e9d16f868ed791d7ac0"
server_key_version: "encryptionkey@0"
I20260810 03:26:34.766391 5663 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786332336319544-28292-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786332336319544-28292-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786332336319544-28292-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:26:34.773943 5445 heartbeater.cc:503] Master 127.27.161.62:36281 was elected leader, sending a full tablet report...
I20260810 03:26:34.788731 5663 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:26:34.793113 5663 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:26:34.793396 5663 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:26:34.794368 5663 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:26:34.795701 5663 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:26:34.795778 5663 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:26:34.795897 5663 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:26:34.795948 5663 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:26:34.825541 5663 rpc_server.cc:307] RPC server started. Bound to: 127.27.161.3:45853
I20260810 03:26:34.825611 5842 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.161.3:45853 every 8 connection(s)
I20260810 03:26:34.826967 5663 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786332336319544-28292-0/minicluster-data/ts-2/data/info.pb
I20260810 03:26:34.830039 28292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iBH5/build/asan/bin/kudu as pid 5663
I20260810 03:26:34.830235 28292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786332336319544-28292-0/minicluster-data/ts-2/wal/instance
I20260810 03:26:34.830729 28292 external_mini_cluster.cc:1428] Setting key ed3c0942017e11438f519a14d918bcad
Aug 10 03:26:34 dist-test-slave-vpvm krb5kdc[5096](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332394, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.161.3@KRBTEST.COM for oryx/127.27.161.62@KRBTEST.COM
I20260810 03:26:34.847097 5845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:34.829823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.161.3:37823 (local address 127.27.161.62:36281)
0810 03:26:34.830350 (+ 527us) server_negotiation.cc:206] Beginning negotiation
0810 03:26:34.830356 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:26:34.831642 (+ 1286us) server_negotiation.cc:407] Connection header received
0810 03:26:34.833298 (+ 1656us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:26:34.833304 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:26:34.833386 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:26:34.833482 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:26:34.835036 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:34.835802 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:34.837312 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:34.837721 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:34.837831 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:34.841435 (+ 3604us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:26:34.841441 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:26:34.841444 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:26:34.841481 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:26:34.843954 (+ 2473us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:26:34.844721 (+ 767us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:26:34.844725 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:26:34.844728 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:26:34.844852 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:26:34.845291 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:26:34.845294 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:26:34.845296 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:26:34.845581 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:26:34.845717 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0810 03:26:34.846480 (+ 763us) server_negotiation.cc:299] Negotiation successful
0810 03:26:34.846715 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":394,"thread_start_us":229,"threads_started":1}
I20260810 03:26:34.848126 5846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:34.830401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:36281 (local address 127.27.161.3:37823)
0810 03:26:34.831244 (+ 843us) negotiation.cc:107] Waiting for socket to connect
0810 03:26:34.831312 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0810 03:26:34.833020 (+ 1708us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:26:34.833746 (+ 726us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:26:34.833754 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:26:34.834304 (+ 550us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:26:34.834864 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:26:34.834882 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:26:34.835952 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:26:34.835956 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:26:34.837169 (+ 1213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:26:34.837182 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:26:34.837972 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:26:34.837982 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:26:34.838220 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:34.838887 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:26:34.838912 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:26:34.841201 (+ 2289us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:26:34.844112 (+ 2911us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:26:34.844123 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:26:34.844139 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:26:34.844592 (+ 453us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:26:34.844958 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:26:34.844963 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:26:34.844966 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:26:34.845169 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:26:34.845711 (+ 542us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:26:34.845719 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:26:34.846102 (+ 383us) client_negotiation.cc:769] Sending connection context
0810 03:26:34.846347 (+ 245us) client_negotiation.cc:240] Negotiation successful
0810 03:26:34.846919 (+ 572us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":560,"thread_start_us":319,"threads_started":1}
I20260810 03:26:34.849516 5843 heartbeater.cc:347] Connected to a master server at 127.27.161.62:36281
I20260810 03:26:34.849812 5843 heartbeater.cc:464] Registering TS with master...
I20260810 03:26:34.850613 5843 heartbeater.cc:511] Master 127.27.161.62:36281 requested a full tablet report, sending...
I20260810 03:26:34.852044 5158 ts_manager.cc:194] Registered new tserver with Master: 05f07b3f00144d75b6c6fc9a40632f17 (127.27.161.3:45853)
I20260810 03:26:34.852864 5158 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.27.161.3@KRBTEST.COM'} at 127.27.161.3:37823
I20260810 03:26:34.857815 28292 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.27.161.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.161.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.161.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786332336319544-28292-0/krb5kdc/oryx_127.27.161.61.keytab.
Entry for principal oryx/127.27.161.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786332336319544-28292-0/krb5kdc/oryx_127.27.161.61.keytab.
I20260810 03:26:34.915822 28292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iBH5/build/asan/bin/kudu
/tmp/dist-test-task24iBH5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786332336319544-28292-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786332336319544-28292-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786332336319544-28292-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786332336319544-28292-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.161.61:43223
--webserver_interface=127.27.161.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786332336319544-28292-0/krb5kdc/oryx_127.27.161.61.keytab
--principal=oryx/127.27.161.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.161.20:43095
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.161.62:36281,127.27.161.61:43223
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786332336319544-28292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786332336319544-28292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786332336319544-28292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:26:35.263957 5853 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:26:35.264322 5853 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:26:35.264463 5853 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:26:35.272233 5853 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:26:35.272380 5853 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:26:35.272464 5853 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:26:35.272544 5853 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:26:35.272605 5853 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:26:35.283779 5853 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.161.20:43095
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786332336319544-28292-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786332336319544-28292-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.27.161.62:36281,127.27.161.61:43223
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786332336319544-28292-0/krb5kdc/oryx_127.27.161.61.keytab
--principal=oryx/127.27.161.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.161.61:43223
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786332336319544-28292-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.27.161.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786332336319544-28292-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
W20260810 03:26:35.284240 5853 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.27.161.62:36281,127.27.161.61:43223'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260810 03:26:35.285413 5853 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:26:35.286895 5853 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:26:35.298899 5862 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:26:35.298936 5859 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:26:35.298858 5865 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:26:35.299461 5853 server_base.cc:1034] running on GCE node
I20260810 03:26:35.300313 5853 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:26:35.301743 5853 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:26:35.303009 5853 hybrid_clock.cc:648] HybridClock initialized: now 1786332395302929 us; error 42 us; skew 500 ppm
I20260810 03:26:35.303400 5853 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:26:35 dist-test-slave-vpvm krb5kdc[5096](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332395, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.161.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:26:35.306607 5853 init.cc:377] Logged in from keytab as oryx/127.27.161.61@KRBTEST.COM (short username oryx)
I20260810 03:26:35.308144 5853 webserver.cc:460] Webserver started at http://127.27.161.61:40251/ using document root <none> and password file <none>
I20260810 03:26:35.308992 5853 fs_manager.cc:359] Metadata directory not provided
I20260810 03:26:35.309106 5853 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:26:35.309485 5853 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:26:35.313086 5853 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786332336319544-28292-0/minicluster-data/master-1/data/instance:
uuid: "d102fadcc4b047ac9e652e1e0a4670a4"
format_stamp: "Formatted at 2026-08-10 03:26:35 on dist-test-slave-vpvm"
server_key: "e45b77885624520f0019cd3f74e6292a"
server_key_iv: "5a89bb200e7574afe9e94537ec9833f5"
server_key_version: "encryptionkey@0"
I20260810 03:26:35.313930 5853 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786332336319544-28292-0/minicluster-data/master-1/wal/instance:
uuid: "d102fadcc4b047ac9e652e1e0a4670a4"
format_stamp: "Formatted at 2026-08-10 03:26:35 on dist-test-slave-vpvm"
server_key: "e45b77885624520f0019cd3f74e6292a"
server_key_iv: "5a89bb200e7574afe9e94537ec9833f5"
server_key_version: "encryptionkey@0"
I20260810 03:26:35.317457 5653 heartbeater.cc:503] Master 127.27.161.62:36281 was elected leader, sending a full tablet report...
I20260810 03:26:35.320513 5853 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.004s
I20260810 03:26:35.324872 5881 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:26:35.326713 5853 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260810 03:26:35.326911 5853 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786332336319544-28292-0/minicluster-data/master-1/data,/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786332336319544-28292-0/minicluster-data/master-1/wal
uuid: "d102fadcc4b047ac9e652e1e0a4670a4"
format_stamp: "Formatted at 2026-08-10 03:26:35 on dist-test-slave-vpvm"
server_key: "e45b77885624520f0019cd3f74e6292a"
server_key_iv: "5a89bb200e7574afe9e94537ec9833f5"
server_key_version: "encryptionkey@0"
I20260810 03:26:35.327147 5853 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786332336319544-28292-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786332336319544-28292-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786332336319544-28292-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:26:35.365715 5853 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:26:35.370496 5853 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:26:35.370779 5853 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:26:35.390789 5853 rpc_server.cc:307] RPC server started. Bound to: 127.27.161.61:43223
I20260810 03:26:35.390823 5957 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.161.61:43223 every 8 connection(s)
I20260810 03:26:35.392668 5853 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786332336319544-28292-0/minicluster-data/master-1/data/info.pb
I20260810 03:26:35.394237 28292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iBH5/build/asan/bin/kudu as pid 5853
I20260810 03:26:35.397554 5962 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:26:35.407071 5962 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.161.62" port: 36281 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 03:26:35 dist-test-slave-vpvm krb5kdc[5096](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332392, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.27.161.61@KRBTEST.COM
Aug 10 03:26:35 dist-test-slave-vpvm krb5kdc[5096](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332395, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.161.61@KRBTEST.COM for oryx/127.27.161.62@KRBTEST.COM
I20260810 03:26:35.421214 5966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:35.396786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50732 (local address 127.27.161.61:43223)
0810 03:26:35.397626 (+ 840us) server_negotiation.cc:206] Beginning negotiation
0810 03:26:35.397644 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0810 03:26:35.397700 (+ 56us) server_negotiation.cc:407] Connection header received
0810 03:26:35.399428 (+ 1728us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:26:35.399453 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:26:35.399884 (+ 431us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:26:35.400244 (+ 360us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:26:35.402102 (+ 1858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:35.403656 (+ 1554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:35.405630 (+ 1974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:35.406398 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:35.406546 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:35.412435 (+ 5889us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:26:35.412445 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:26:35.412456 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:26:35.412510 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:26:35.415898 (+ 3388us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:26:35.416803 (+ 905us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:26:35.416810 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:26:35.416818 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:26:35.417062 (+ 244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:26:35.417479 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:26:35.417483 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:26:35.417485 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:26:35.418349 (+ 864us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:26:35.418683 (+ 334us) server_negotiation.cc:1035] Waiting for connection context
0810 03:26:35.419201 (+ 518us) server_negotiation.cc:299] Negotiation successful
0810 03:26:35.419775 (+ 574us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":604,"thread_start_us":355,"threads_started":1}
I20260810 03:26:35.427477 5968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:35.410459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:36281 (local address 127.0.0.1:54756)
0810 03:26:35.411205 (+ 746us) negotiation.cc:107] Waiting for socket to connect
0810 03:26:35.411254 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0810 03:26:35.411415 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:26:35.412156 (+ 741us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:26:35.412164 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:26:35.412919 (+ 755us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:26:35.413257 (+ 338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:26:35.413273 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:26:35.414844 (+ 1571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:26:35.414848 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:26:35.416013 (+ 1165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:26:35.416026 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:26:35.417004 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:26:35.417011 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:26:35.417199 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:35.417795 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:26:35.417817 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:26:35.420927 (+ 3110us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:26:35.424384 (+ 3457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:26:35.424395 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:26:35.424402 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:26:35.424865 (+ 463us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:26:35.425217 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:26:35.425220 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:26:35.425222 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:26:35.425369 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:26:35.426418 (+ 1049us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:26:35.426424 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:26:35.426632 (+ 208us) client_negotiation.cc:769] Sending connection context
0810 03:26:35.426879 (+ 247us) client_negotiation.cc:240] Negotiation successful
0810 03:26:35.427174 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":308,"threads_started":1}
I20260810 03:26:35.427523 5969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:35.410574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54756 (local address 127.27.161.62:36281)
0810 03:26:35.411791 (+ 1217us) server_negotiation.cc:206] Beginning negotiation
0810 03:26:35.411796 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:26:35.411816 (+ 20us) server_negotiation.cc:407] Connection header received
0810 03:26:35.411874 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:26:35.411878 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:26:35.411949 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:26:35.412046 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:26:35.413468 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:35.414673 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:35.416159 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:35.416799 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:35.416904 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:35.421193 (+ 4289us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:26:35.421200 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:26:35.421203 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:26:35.421250 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:26:35.424181 (+ 2931us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:26:35.424990 (+ 809us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:26:35.424994 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:26:35.424997 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:26:35.425130 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:26:35.425632 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:26:35.425642 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:26:35.425645 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:26:35.426279 (+ 634us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:26:35.426411 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0810 03:26:35.427043 (+ 632us) server_negotiation.cc:299] Negotiation successful
0810 03:26:35.427258 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1109,"thread_start_us":1001,"threads_started":1}
I20260810 03:26:35.429648 5962 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.161.61" port: 43223 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 03:26:35 dist-test-slave-vpvm krb5kdc[5096](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332395, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.161.61@KRBTEST.COM for oryx/127.27.161.61@KRBTEST.COM
I20260810 03:26:35.443846 5968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:35.430392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.61:43223 (local address 127.0.0.1:50742)
0810 03:26:35.430808 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0810 03:26:35.430899 (+ 91us) client_negotiation.cc:174] Beginning negotiation
0810 03:26:35.431028 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:26:35.431320 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:26:35.431323 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:26:35.431792 (+ 469us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:26:35.431962 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:26:35.431971 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:26:35.433021 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:26:35.433025 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:26:35.433911 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:26:35.433925 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:26:35.434707 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:26:35.434711 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:26:35.434866 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:35.435470 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:26:35.435488 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:26:35.437817 (+ 2329us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:26:35.440962 (+ 3145us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:26:35.440967 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:26:35.440971 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:26:35.441370 (+ 399us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:26:35.442119 (+ 749us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:26:35.442123 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:26:35.442126 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:26:35.442282 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:26:35.443007 (+ 725us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:26:35.443011 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:26:35.443171 (+ 160us) client_negotiation.cc:769] Sending connection context
0810 03:26:35.443313 (+ 142us) client_negotiation.cc:240] Negotiation successful
0810 03:26:35.443562 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148}
I20260810 03:26:35.444115 5966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:35.430584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50742 (local address 127.27.161.61:43223)
0810 03:26:35.430782 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0810 03:26:35.430788 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:26:35.430998 (+ 210us) server_negotiation.cc:407] Connection header received
0810 03:26:35.431126 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:26:35.431131 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:26:35.431210 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:26:35.431296 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:26:35.432084 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:35.432868 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:35.434093 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:35.434549 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:35.434659 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:35.438082 (+ 3423us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:26:35.438088 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:26:35.438091 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:26:35.438135 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:26:35.440766 (+ 2631us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:26:35.441617 (+ 851us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:26:35.441625 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:26:35.441628 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:26:35.441922 (+ 294us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:26:35.442444 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:26:35.442450 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:26:35.442452 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:26:35.442855 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:26:35.442990 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0810 03:26:35.443491 (+ 501us) server_negotiation.cc:299] Negotiation successful
0810 03:26:35.443724 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260810 03:26:35.448987 5962 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d102fadcc4b047ac9e652e1e0a4670a4: Bootstrap starting.
I20260810 03:26:35.452275 5962 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d102fadcc4b047ac9e652e1e0a4670a4: Neither blocks nor log segments found. Creating new log.
I20260810 03:26:35.453398 5962 log.cc:824] T 00000000000000000000000000000000 P d102fadcc4b047ac9e652e1e0a4670a4: Log is configured to *not* fsync() on all Append() calls
I20260810 03:26:35.456570 5962 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d102fadcc4b047ac9e652e1e0a4670a4: No bootstrap required, opened a new log
I20260810 03:26:35.464051 5962 raft_consensus.cc:348] T 00000000000000000000000000000000 P d102fadcc4b047ac9e652e1e0a4670a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b65187fc20346eda207dd58ee639ef2" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 36281 } } peers { permanent_uuid: "d102fadcc4b047ac9e652e1e0a4670a4" member_type: VOTER last_known_addr { host: "127.27.161.61" port: 43223 } }
I20260810 03:26:35.464453 5962 raft_consensus.cc:374] T 00000000000000000000000000000000 P d102fadcc4b047ac9e652e1e0a4670a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:26:35.464560 5962 raft_consensus.cc:729] T 00000000000000000000000000000000 P d102fadcc4b047ac9e652e1e0a4670a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d102fadcc4b047ac9e652e1e0a4670a4, State: Initialized, Role: FOLLOWER
I20260810 03:26:35.465171 5962 consensus_queue.cc:260] T 00000000000000000000000000000000 P d102fadcc4b047ac9e652e1e0a4670a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b65187fc20346eda207dd58ee639ef2" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 36281 } } peers { permanent_uuid: "d102fadcc4b047ac9e652e1e0a4670a4" member_type: VOTER last_known_addr { host: "127.27.161.61" port: 43223 } }
I20260810 03:26:35.466136 5971 sys_catalog.cc:455] T 00000000000000000000000000000000 P d102fadcc4b047ac9e652e1e0a4670a4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b65187fc20346eda207dd58ee639ef2" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 36281 } } peers { permanent_uuid: "d102fadcc4b047ac9e652e1e0a4670a4" member_type: VOTER last_known_addr { host: "127.27.161.61" port: 43223 } } }
I20260810 03:26:35.466466 5971 sys_catalog.cc:458] T 00000000000000000000000000000000 P d102fadcc4b047ac9e652e1e0a4670a4 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 03:26:35.467283 5962 sys_catalog.cc:564] T 00000000000000000000000000000000 P d102fadcc4b047ac9e652e1e0a4670a4 [sys.catalog]: configured and running, proceeding with master startup.
W20260810 03:26:35.478193 5987 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d102fadcc4b047ac9e652e1e0a4670a4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 03:26:35.478477 5987 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 03:26:35.482651 5853 master_runner.cc:428] Detected that this master d102fadcc4b047ac9e652e1e0a4670a4 is joining an existing cluster
I20260810 03:26:35.482791 5853 master_runner.cc:432] Initiating AddMaster RPC to add 127.27.161.61:43223
I20260810 03:26:35.506884 5969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:35.487933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54768 (local address 127.27.161.62:36281)
0810 03:26:35.488153 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0810 03:26:35.488160 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:26:35.488865 (+ 705us) server_negotiation.cc:407] Connection header received
0810 03:26:35.489215 (+ 350us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:26:35.489223 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:26:35.489316 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:26:35.489398 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:26:35.491178 (+ 1780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:35.492275 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:35.495483 (+ 3208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:35.496205 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:35.496326 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:35.499304 (+ 2978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:26:35.499313 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:26:35.499318 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:26:35.499369 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:26:35.503256 (+ 3887us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:26:35.504117 (+ 861us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:26:35.504123 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:26:35.504126 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:26:35.504289 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:26:35.504782 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:26:35.504788 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:26:35.504791 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:26:35.505169 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:26:35.505317 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0810 03:26:35.506272 (+ 955us) server_negotiation.cc:299] Negotiation successful
0810 03:26:35.506525 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260810 03:26:35.507956 5996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:35.487971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:36281 (local address 127.0.0.1:54768)
0810 03:26:35.488732 (+ 761us) negotiation.cc:107] Waiting for socket to connect
0810 03:26:35.488756 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0810 03:26:35.489027 (+ 271us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:26:35.489662 (+ 635us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:26:35.489667 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:26:35.490488 (+ 821us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:26:35.490957 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:26:35.490984 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:26:35.492496 (+ 1512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:26:35.492502 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:26:35.495268 (+ 2766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:26:35.495288 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:26:35.496700 (+ 1412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:26:35.496708 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:26:35.496923 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:35.497740 (+ 817us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:26:35.497770 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:26:35.499017 (+ 1247us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:26:35.503428 (+ 4411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:26:35.503433 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:26:35.503437 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:26:35.503964 (+ 527us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:26:35.504444 (+ 480us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:26:35.504448 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:26:35.504452 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:26:35.504645 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:26:35.505398 (+ 753us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:26:35.505402 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:26:35.505659 (+ 257us) client_negotiation.cc:769] Sending connection context
0810 03:26:35.507094 (+ 1435us) client_negotiation.cc:240] Negotiation successful
0810 03:26:35.507437 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":559,"mutex_wait_us":2,"thread_start_us":373,"threads_started":1}
I20260810 03:26:35.514642 5966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:35.489201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50748 (local address 127.27.161.61:43223)
0810 03:26:35.489367 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0810 03:26:35.489373 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:26:35.500380 (+ 11007us) server_negotiation.cc:407] Connection header received
0810 03:26:35.500610 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:26:35.500615 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:26:35.500687 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:26:35.500764 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:26:35.501813 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:35.502564 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:35.504113 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:35.504793 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:35.504881 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:35.507340 (+ 2459us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:26:35.507346 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:26:35.507351 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:26:35.507390 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:26:35.511775 (+ 4385us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:26:35.512488 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:26:35.512494 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:26:35.512499 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:26:35.512669 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:26:35.513084 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:26:35.513089 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:26:35.513093 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:26:35.513387 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:26:35.513521 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0810 03:26:35.513997 (+ 476us) server_negotiation.cc:299] Negotiation successful
0810 03:26:35.514338 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":141,"server-negotiator.queue_time_us":57}
I20260810 03:26:35.515182 6001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:35.488889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.61:43223 (local address 127.0.0.1:50748)
0810 03:26:35.500282 (+ 11393us) negotiation.cc:107] Waiting for socket to connect
0810 03:26:35.500303 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0810 03:26:35.500478 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:26:35.500809 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:26:35.500813 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:26:35.501337 (+ 524us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:26:35.501686 (+ 349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:26:35.501697 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:26:35.502721 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:26:35.502726 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:26:35.503982 (+ 1256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:26:35.503994 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:26:35.505219 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:26:35.505229 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:26:35.505406 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:35.506069 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:26:35.506091 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:26:35.507158 (+ 1067us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:26:35.511940 (+ 4782us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:26:35.511945 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:26:35.511949 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:26:35.512369 (+ 420us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:26:35.512778 (+ 409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:26:35.512783 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:26:35.512786 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:26:35.512950 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:26:35.513502 (+ 552us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:26:35.513506 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:26:35.513641 (+ 135us) client_negotiation.cc:769] Sending connection context
0810 03:26:35.513764 (+ 123us) client_negotiation.cc:240] Negotiation successful
0810 03:26:35.514810 (+ 1046us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11258,"mutex_wait_us":26,"thread_start_us":10991,"threads_started":1}
Aug 10 03:26:35 dist-test-slave-vpvm krb5kdc[5096](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332392, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.161.62@KRBTEST.COM for oryx/127.27.161.61@KRBTEST.COM
WARNING: no policy specified for oryx/127.27.161.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.161.60@KRBTEST.COM" created.
I20260810 03:26:35.533057 5966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:35.516973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50764 (local address 127.27.161.61:43223)
0810 03:26:35.517744 (+ 771us) server_negotiation.cc:206] Beginning negotiation
0810 03:26:35.517749 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:26:35.517762 (+ 13us) server_negotiation.cc:407] Connection header received
0810 03:26:35.517830 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:26:35.517833 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:26:35.517912 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:26:35.518022 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:26:35.519319 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:35.520098 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:35.521597 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:35.522117 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:35.522253 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:35.525510 (+ 3257us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:26:35.525515 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:26:35.525518 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:26:35.525553 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:26:35.528529 (+ 2976us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:26:35.529646 (+ 1117us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:26:35.529651 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:26:35.529653 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:26:35.529831 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:26:35.530361 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:26:35.530367 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:26:35.530369 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:26:35.530805 (+ 436us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:26:35.532302 (+ 1497us) server_negotiation.cc:1035] Waiting for connection context
0810 03:26:35.532457 (+ 155us) server_negotiation.cc:299] Negotiation successful
0810 03:26:35.532709 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":679}
I20260810 03:26:35.533555 6004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:35.516501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.61:43223 (local address 127.0.0.1:50764)
0810 03:26:35.517131 (+ 630us) negotiation.cc:107] Waiting for socket to connect
0810 03:26:35.517187 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0810 03:26:35.517367 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:26:35.518107 (+ 740us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:26:35.518116 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:26:35.518885 (+ 769us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:26:35.519126 (+ 241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:26:35.519137 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:26:35.520245 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:26:35.520249 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:26:35.521363 (+ 1114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:26:35.521377 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:26:35.522285 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:26:35.522292 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:26:35.522488 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:35.523099 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:26:35.523116 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:26:35.525317 (+ 2201us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:26:35.528709 (+ 3392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:26:35.528729 (+ 20us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:26:35.528739 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:26:35.529453 (+ 714us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:26:35.530034 (+ 581us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:26:35.530038 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:26:35.530041 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:26:35.530196 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:26:35.531125 (+ 929us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:26:35.531135 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:26:35.531404 (+ 269us) client_negotiation.cc:769] Sending connection context
0810 03:26:35.531638 (+ 234us) client_negotiation.cc:240] Negotiation successful
0810 03:26:35.532019 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":510,"threads_started":1}
I20260810 03:26:35.535975 5158 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.27.161.61:43223: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "d102fadcc4b047ac9e652e1e0a4670a4"
member_type: NON_VOTER
last_known_addr {
host: "127.27.161.61"
port: 43223
}
attrs {
promote: true
}
}
dest_uuid: "4b65187fc20346eda207dd58ee639ef2"
cas_config_opid_index: -1
I20260810 03:26:35.537609 5158 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4b65187fc20346eda207dd58ee639ef2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4b65187fc20346eda207dd58ee639ef2" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 36281 } } peers { permanent_uuid: "d102fadcc4b047ac9e652e1e0a4670a4" member_type: NON_VOTER last_known_addr { host: "127.27.161.61" port: 43223 } attrs { promote: true } }
I20260810 03:26:35.544818 6008 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 4b65187fc20346eda207dd58ee639ef2 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER d102fadcc4b047ac9e652e1e0a4670a4 (127.27.161.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4b65187fc20346eda207dd58ee639ef2" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 36281 } } peers { permanent_uuid: "d102fadcc4b047ac9e652e1e0a4670a4" member_type: NON_VOTER last_known_addr { host: "127.27.161.61" port: 43223 } attrs { promote: true } } }
I20260810 03:26:35.546147 5927 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d102fadcc4b047ac9e652e1e0a4670a4 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:26:35.549741 5927 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d102fadcc4b047ac9e652e1e0a4670a4 [term 1 FOLLOWER]: Refusing update from remote peer 4b65187fc20346eda207dd58ee639ef2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260810 03:26:35.549834 6008 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.27.161.61:43223
I20260810 03:26:35.550872 5853 master.cc:537] Master@127.27.161.61:43223 shutting down...
I20260810 03:26:35.551656 6008 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4b65187fc20346eda207dd58ee639ef2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d102fadcc4b047ac9e652e1e0a4670a4" member_type: NON_VOTER last_known_addr { host: "127.27.161.61" port: 43223 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:26:35.552105 6006 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b65187fc20346eda207dd58ee639ef2 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "4b65187fc20346eda207dd58ee639ef2" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4b65187fc20346eda207dd58ee639ef2" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 36281 } } peers { permanent_uuid: "d102fadcc4b047ac9e652e1e0a4670a4" member_type: NON_VOTER last_known_addr { host: "127.27.161.61" port: 43223 } attrs { promote: true } } }
I20260810 03:26:35.552647 6006 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b65187fc20346eda207dd58ee639ef2 [sys.catalog]: This master's current role is: LEADER
I20260810 03:26:35.565004 5853 raft_consensus.cc:2229] T 00000000000000000000000000000000 P d102fadcc4b047ac9e652e1e0a4670a4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 03:26:35.565599 5853 raft_consensus.cc:2258] T 00000000000000000000000000000000 P d102fadcc4b047ac9e652e1e0a4670a4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:26:35.565896 5853 tablet_replica.cc:330] T 00000000000000000000000000000000 P d102fadcc4b047ac9e652e1e0a4670a4: stopping tablet replica
W20260810 03:26:35.568112 5135 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260810 03:26:35.570009 5135 consensus_peers.cc:487] T 00000000000000000000000000000000 P 4b65187fc20346eda207dd58ee639ef2 -> Peer d102fadcc4b047ac9e652e1e0a4670a4 (127.27.161.61:43223): Couldn't send request to peer d102fadcc4b047ac9e652e1e0a4670a4. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.161.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786332336319544-28292-0/krb5kdc/oryx_127.27.161.60.keytab.
Entry for principal oryx/127.27.161.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786332336319544-28292-0/krb5kdc/oryx_127.27.161.60.keytab.
I20260810 03:26:35.577760 28292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iBH5/build/asan/bin/kudu
/tmp/dist-test-task24iBH5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786332336319544-28292-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786332336319544-28292-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786332336319544-28292-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786332336319544-28292-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.161.60:43217
--webserver_interface=127.27.161.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786332336319544-28292-0/krb5kdc/oryx_127.27.161.60.keytab
--principal=oryx/127.27.161.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.161.20:43095
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.161.62:36281,127.27.161.61:43223,127.27.161.60:43217
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786332336319544-28292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786332336319544-28292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786332336319544-28292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 03:26:35.584419 5853 master.cc:552] Master@127.27.161.61:43223 shutdown complete.
I20260810 03:26:35.584565 5853 master_runner.cc:305] Clearing existing system tablet
I20260810 03:26:35.585013 5853 fs_manager.cc:359] Metadata directory not provided
I20260810 03:26:35.585111 5853 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:26:35.587604 5853 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.004s
I20260810 03:26:35.589268 6017 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:26:35.589735 5853 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786332336319544-28292-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:26:35.590027 5853 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 03:26:35.590123 5853 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786332336319544-28292-0/minicluster-data/master-1/data,/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786332336319544-28292-0/minicluster-data/master-1/wal
uuid: "d102fadcc4b047ac9e652e1e0a4670a4"
format_stamp: "Formatted at 2026-08-10 03:26:35 on dist-test-slave-vpvm"
server_key: "e45b77885624520f0019cd3f74e6292a"
server_key_iv: "5a89bb200e7574afe9e94537ec9833f5"
server_key_version: "encryptionkey@0"
I20260810 03:26:35.595503 5853 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P d102fadcc4b047ac9e652e1e0a4670a4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:26:35.602453 5853 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P d102fadcc4b047ac9e652e1e0a4670a4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260810 03:26:35.602682 5853 log.cc:1196] T 00000000000000000000000000000000 P d102fadcc4b047ac9e652e1e0a4670a4: Deleting WAL directory at /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786332336319544-28292-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260810 03:26:35.603183 5853 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P d102fadcc4b047ac9e652e1e0a4670a4: Deleting consensus metadata
I20260810 03:26:35.603519 5853 master_runner.cc:315] Copying system tablet from 127.27.161.62:36281
I20260810 03:26:35.606746 5853 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P d102fadcc4b047ac9e652e1e0a4670a4: tablet copy: Beginning tablet copy session from remote peer at address 127.27.161.62:36281
I20260810 03:26:35.631474 5969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:35.608014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54784 (local address 127.27.161.62:36281)
0810 03:26:35.608233 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0810 03:26:35.608240 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:26:35.612699 (+ 4459us) server_negotiation.cc:407] Connection header received
0810 03:26:35.612909 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:26:35.612916 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:26:35.613008 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:26:35.613134 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:26:35.614307 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:35.615451 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:35.617435 (+ 1984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:35.618219 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:35.618356 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:35.624533 (+ 6177us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:26:35.624541 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:26:35.624546 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:26:35.624597 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:26:35.628251 (+ 3654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:26:35.629175 (+ 924us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:26:35.629181 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:26:35.629185 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:26:35.629346 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:26:35.629779 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:26:35.629784 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:26:35.629788 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:26:35.630221 (+ 433us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:26:35.630378 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0810 03:26:35.630853 (+ 475us) server_negotiation.cc:299] Negotiation successful
0810 03:26:35.631091 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260810 03:26:35.631685 6026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:35.607924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:36281 (local address 127.0.0.1:54784)
0810 03:26:35.612543 (+ 4619us) negotiation.cc:107] Waiting for socket to connect
0810 03:26:35.612569 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0810 03:26:35.612759 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:26:35.613177 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:26:35.613183 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:26:35.613757 (+ 574us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:26:35.614136 (+ 379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:26:35.614150 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:26:35.615619 (+ 1469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:26:35.615625 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:26:35.617250 (+ 1625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:26:35.617268 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:26:35.618390 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:26:35.618397 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:26:35.618587 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:35.623004 (+ 4417us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:26:35.623037 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:26:35.624227 (+ 1190us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:26:35.628431 (+ 4204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:26:35.628437 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:26:35.628441 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:26:35.629035 (+ 594us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:26:35.629470 (+ 435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:26:35.629475 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:26:35.629479 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:26:35.629663 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:26:35.630350 (+ 687us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:26:35.630355 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:26:35.630564 (+ 209us) client_negotiation.cc:769] Sending connection context
0810 03:26:35.630721 (+ 157us) client_negotiation.cc:240] Negotiation successful
0810 03:26:35.631157 (+ 436us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4443,"thread_start_us":4329,"threads_started":1}
I20260810 03:26:35.632779 5190 tablet_copy_service.cc:140] P 4b65187fc20346eda207dd58ee639ef2: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer d102fadcc4b047ac9e652e1e0a4670a4 ({username='oryx', principal='oryx/127.27.161.61@KRBTEST.COM'} at 127.0.0.1:54784)
I20260810 03:26:35.633253 5190 tablet_copy_service.cc:161] P 4b65187fc20346eda207dd58ee639ef2: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer d102fadcc4b047ac9e652e1e0a4670a4 at {username='oryx', principal='oryx/127.27.161.61@KRBTEST.COM'} at 127.0.0.1:54784: session id = d102fadcc4b047ac9e652e1e0a4670a4-00000000000000000000000000000000
I20260810 03:26:35.638192 5190 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 4b65187fc20346eda207dd58ee639ef2: Tablet Copy: opened 0 blocks and 1 log segments
I20260810 03:26:35.641201 5853 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260810 03:26:35.648492 5853 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P d102fadcc4b047ac9e652e1e0a4670a4: tablet copy: Starting download of 0 data blocks...
I20260810 03:26:35.648932 5853 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P d102fadcc4b047ac9e652e1e0a4670a4: tablet copy: Starting download of 1 WAL segments...
I20260810 03:26:35.652987 5853 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P d102fadcc4b047ac9e652e1e0a4670a4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260810 03:26:35.658012 5190 tablet_copy_service.cc:342] P 4b65187fc20346eda207dd58ee639ef2: Request end of tablet copy session d102fadcc4b047ac9e652e1e0a4670a4-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.27.161.61@KRBTEST.COM'} at 127.0.0.1:54784
I20260810 03:26:35.658286 5190 tablet_copy_service.cc:433] P 4b65187fc20346eda207dd58ee639ef2: ending tablet copy session d102fadcc4b047ac9e652e1e0a4670a4-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer d102fadcc4b047ac9e652e1e0a4670a4
I20260810 03:26:35.671656 5853 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:26:35.674911 6028 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:26:35.675359 6029 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:26:35.683310 5853 server_base.cc:1034] running on GCE node
W20260810 03:26:35.686367 6032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:26:35.686913 5853 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:26:35.687661 5853 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:26:35.688881 5853 hybrid_clock.cc:648] HybridClock initialized: now 1786332395688813 us; error 43 us; skew 500 ppm
I20260810 03:26:35.689124 5853 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:26:35 dist-test-slave-vpvm krb5kdc[5096](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332395, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.161.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:26:35.691197 5853 init.cc:377] Logged in from keytab as oryx/127.27.161.61@KRBTEST.COM (short username oryx)
I20260810 03:26:35.694762 5853 webserver.cc:460] Webserver started at http://127.27.161.61:39417/ using document root <none> and password file <none>
I20260810 03:26:35.695127 5853 fs_manager.cc:359] Metadata directory not provided
I20260810 03:26:35.695210 5853 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:26:35.698419 5853 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.001s
I20260810 03:26:35.700563 6044 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:26:35.701007 5853 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 03:26:35.701205 5853 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786332336319544-28292-0/minicluster-data/master-1/data,/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786332336319544-28292-0/minicluster-data/master-1/wal
uuid: "d102fadcc4b047ac9e652e1e0a4670a4"
format_stamp: "Formatted at 2026-08-10 03:26:35 on dist-test-slave-vpvm"
server_key: "e45b77885624520f0019cd3f74e6292a"
server_key_iv: "5a89bb200e7574afe9e94537ec9833f5"
server_key_version: "encryptionkey@0"
I20260810 03:26:35.701568 5853 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786332336319544-28292-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786332336319544-28292-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786332336319544-28292-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:26:35.711453 5853 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:26:35.716616 5853 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:26:35.747869 5853 rpc_server.cc:307] RPC server started. Bound to: 127.27.161.61:43223
I20260810 03:26:35.747924 6121 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.161.61:43223 every 8 connection(s)
I20260810 03:26:35.749154 5853 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786332336319544-28292-0/minicluster-data/master-1/data/info.pb
I20260810 03:26:35.754580 6122 sys_catalog.cc:263] Verifying existing consensus state
I20260810 03:26:35.758754 6122 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d102fadcc4b047ac9e652e1e0a4670a4: Bootstrap starting.
I20260810 03:26:35.779296 6122 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d102fadcc4b047ac9e652e1e0a4670a4: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 03:26:35.779934 6122 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d102fadcc4b047ac9e652e1e0a4670a4: Bootstrap complete.
I20260810 03:26:35.781098 6122 raft_consensus.cc:348] T 00000000000000000000000000000000 P d102fadcc4b047ac9e652e1e0a4670a4 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4b65187fc20346eda207dd58ee639ef2" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 36281 } } peers { permanent_uuid: "d102fadcc4b047ac9e652e1e0a4670a4" member_type: NON_VOTER last_known_addr { host: "127.27.161.61" port: 43223 } attrs { promote: true } }
I20260810 03:26:35.781385 6122 raft_consensus.cc:729] T 00000000000000000000000000000000 P d102fadcc4b047ac9e652e1e0a4670a4 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d102fadcc4b047ac9e652e1e0a4670a4, State: Initialized, Role: LEARNER
I20260810 03:26:35.781716 6122 consensus_queue.cc:260] T 00000000000000000000000000000000 P d102fadcc4b047ac9e652e1e0a4670a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4b65187fc20346eda207dd58ee639ef2" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 36281 } } peers { permanent_uuid: "d102fadcc4b047ac9e652e1e0a4670a4" member_type: NON_VOTER last_known_addr { host: "127.27.161.61" port: 43223 } attrs { promote: true } }
I20260810 03:26:35.782670 6125 sys_catalog.cc:455] T 00000000000000000000000000000000 P d102fadcc4b047ac9e652e1e0a4670a4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4b65187fc20346eda207dd58ee639ef2" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 36281 } } peers { permanent_uuid: "d102fadcc4b047ac9e652e1e0a4670a4" member_type: NON_VOTER last_known_addr { host: "127.27.161.61" port: 43223 } attrs { promote: true } } }
I20260810 03:26:35.783100 6122 sys_catalog.cc:564] T 00000000000000000000000000000000 P d102fadcc4b047ac9e652e1e0a4670a4 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:26:35.783144 6125 sys_catalog.cc:458] T 00000000000000000000000000000000 P d102fadcc4b047ac9e652e1e0a4670a4 [sys.catalog]: This master's current role is: LEARNER
I20260810 03:26:35.795310 6139 catalog_manager.cc:1259] Loaded cluster ID: 75ff020955394fb1943d772cebcad889
I20260810 03:26:35.795552 6139 catalog_manager.cc:1552] T 00000000000000000000000000000000 P d102fadcc4b047ac9e652e1e0a4670a4: loading cluster ID for follower catalog manager: success
I20260810 03:26:35.799268 6139 catalog_manager.cc:1574] T 00000000000000000000000000000000 P d102fadcc4b047ac9e652e1e0a4670a4: acquiring CA information for follower catalog manager: success
I20260810 03:26:35.801965 6139 catalog_manager.cc:1602] T 00000000000000000000000000000000 P d102fadcc4b047ac9e652e1e0a4670a4: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 03:26:35.855607 5843 heartbeater.cc:503] Master 127.27.161.62:36281 was elected leader, sending a full tablet report...
W20260810 03:26:35.987418 6015 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:26:35.987740 6015 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:26:35.987871 6015 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:26:35.996469 6015 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:26:35.996678 6015 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:26:35.996783 6015 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:26:35.996865 6015 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:26:35.996924 6015 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:26:35.998914 6004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:35.992091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.61:43223 (local address 127.0.0.1:50774)
0810 03:26:35.992304 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0810 03:26:35.992326 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0810 03:26:35.992467 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:26:35.993365 (+ 898us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:26:35.993369 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:26:35.993401 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:26:35.993583 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:26:35.993593 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:26:35.995109 (+ 1516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:26:35.995113 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:26:35.996531 (+ 1418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:26:35.996546 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:26:35.998056 (+ 1510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:26:35.998062 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:26:35.998347 (+ 285us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:35.998396 (+ 49us) client_negotiation.cc:769] Sending connection context
0810 03:26:35.998489 (+ 93us) client_negotiation.cc:240] Negotiation successful
0810 03:26:35.998637 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20260810 03:26:35.998984 6146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:35.992282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50774 (local address 127.27.161.61:43223)
0810 03:26:35.993021 (+ 739us) server_negotiation.cc:206] Beginning negotiation
0810 03:26:35.993026 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:26:35.993041 (+ 15us) server_negotiation.cc:407] Connection header received
0810 03:26:35.993103 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:26:35.993107 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:26:35.993178 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:26:35.993289 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:26:35.993752 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:35.994950 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:35.996690 (+ 1740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:35.997839 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:35.997985 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:35.998254 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0810 03:26:35.998541 (+ 287us) server_negotiation.cc:299] Negotiation successful
0810 03:26:35.998676 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":628,"thread_start_us":471,"threads_started":1}
I20260810 03:26:36.000409 6083 raft_consensus.cc:1203] T 00000000000000000000000000000000 P d102fadcc4b047ac9e652e1e0a4670a4 [term 1 LEARNER]: Deduplicated request from leader. Original: 0.0->[1.1-1.5] Dedup: 1.5->[]
I20260810 03:26:36.001158 6125 sys_catalog.cc:455] T 00000000000000000000000000000000 P d102fadcc4b047ac9e652e1e0a4670a4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4b65187fc20346eda207dd58ee639ef2. Latest consensus state: current_term: 1 leader_uuid: "4b65187fc20346eda207dd58ee639ef2" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4b65187fc20346eda207dd58ee639ef2" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 36281 } } peers { permanent_uuid: "d102fadcc4b047ac9e652e1e0a4670a4" member_type: NON_VOTER last_known_addr { host: "127.27.161.61" port: 43223 } attrs { promote: true } } }
I20260810 03:26:36.001448 6125 sys_catalog.cc:458] T 00000000000000000000000000000000 P d102fadcc4b047ac9e652e1e0a4670a4 [sys.catalog]: This master's current role is: LEARNER
I20260810 03:26:36.008769 6015 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.161.20:43095
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786332336319544-28292-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786332336319544-28292-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.27.161.62:36281,127.27.161.61:43223,127.27.161.60:43217
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786332336319544-28292-0/krb5kdc/oryx_127.27.161.60.keytab
--principal=oryx/127.27.161.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.161.60:43217
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786332336319544-28292-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.27.161.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786332336319544-28292-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:26:36.010277 6015 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:26:36.011543 6015 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:26:36.025393 6150 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:26:36.025626 6151 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:26:36.025681 6153 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:26:36.026788 6015 server_base.cc:1034] running on GCE node
I20260810 03:26:36.027724 6015 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:26:36.029371 6015 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:26:36.030575 6015 hybrid_clock.cc:648] HybridClock initialized: now 1786332396030467 us; error 95 us; skew 500 ppm
I20260810 03:26:36.030963 6015 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:26:36 dist-test-slave-vpvm krb5kdc[5096](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332396, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.161.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:26:36.034282 6015 init.cc:377] Logged in from keytab as oryx/127.27.161.60@KRBTEST.COM (short username oryx)
I20260810 03:26:36.035880 6015 webserver.cc:460] Webserver started at http://127.27.161.60:44035/ using document root <none> and password file <none>
I20260810 03:26:36.036813 6015 fs_manager.cc:359] Metadata directory not provided
I20260810 03:26:36.036953 6015 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:26:36.037321 6015 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:26:36.040612 6015 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786332336319544-28292-0/minicluster-data/master-2/data/instance:
uuid: "1245ea4d6f734899a3087452b0495ee2"
format_stamp: "Formatted at 2026-08-10 03:26:36 on dist-test-slave-vpvm"
server_key: "73691ae5721db7f7d304a0e61ed586ba"
server_key_iv: "6ebc903fbb147ef98be58a062654bcf0"
server_key_version: "encryptionkey@0"
I20260810 03:26:36.041397 6015 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786332336319544-28292-0/minicluster-data/master-2/wal/instance:
uuid: "1245ea4d6f734899a3087452b0495ee2"
format_stamp: "Formatted at 2026-08-10 03:26:36 on dist-test-slave-vpvm"
server_key: "73691ae5721db7f7d304a0e61ed586ba"
server_key_iv: "6ebc903fbb147ef98be58a062654bcf0"
server_key_version: "encryptionkey@0"
I20260810 03:26:36.047210 6015 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.003s
I20260810 03:26:36.050850 6163 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:26:36.052197 6015 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 03:26:36.052366 6015 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786332336319544-28292-0/minicluster-data/master-2/data,/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786332336319544-28292-0/minicluster-data/master-2/wal
uuid: "1245ea4d6f734899a3087452b0495ee2"
format_stamp: "Formatted at 2026-08-10 03:26:36 on dist-test-slave-vpvm"
server_key: "73691ae5721db7f7d304a0e61ed586ba"
server_key_iv: "6ebc903fbb147ef98be58a062654bcf0"
server_key_version: "encryptionkey@0"
I20260810 03:26:36.052546 6015 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786332336319544-28292-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786332336319544-28292-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786332336319544-28292-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:26:36.068567 6015 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:26:36.073294 6015 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:26:36.073577 6015 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:26:36.103440 6015 rpc_server.cc:307] RPC server started. Bound to: 127.27.161.60:43217
I20260810 03:26:36.103567 6237 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.161.60:43217 every 8 connection(s)
I20260810 03:26:36.105252 6015 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786332336319544-28292-0/minicluster-data/master-2/data/info.pb
I20260810 03:26:36.109808 6239 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:26:36.111963 28292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iBH5/build/asan/bin/kudu as pid 6015
I20260810 03:26:36.120711 6239 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.161.62" port: 36281 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 03:26:36 dist-test-slave-vpvm krb5kdc[5096](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332392, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.27.161.60@KRBTEST.COM
Aug 10 03:26:36 dist-test-slave-vpvm krb5kdc[5096](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332396, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.161.60@KRBTEST.COM for oryx/127.27.161.62@KRBTEST.COM
I20260810 03:26:36.140571 6242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:36.114306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46896 (local address 127.27.161.60:43217)
0810 03:26:36.115454 (+ 1148us) server_negotiation.cc:206] Beginning negotiation
0810 03:26:36.115467 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 03:26:36.115524 (+ 57us) server_negotiation.cc:407] Connection header received
0810 03:26:36.117649 (+ 2125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:26:36.117677 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:26:36.118250 (+ 573us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:26:36.118648 (+ 398us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:26:36.121061 (+ 2413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:36.122554 (+ 1493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:36.124386 (+ 1832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:36.125544 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:36.125690 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:36.131592 (+ 5902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:26:36.131603 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:26:36.131618 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:26:36.131667 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:26:36.135312 (+ 3645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:26:36.136222 (+ 910us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:26:36.136231 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:26:36.136242 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:26:36.136458 (+ 216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:26:36.136970 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:26:36.136975 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:26:36.136977 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:26:36.137720 (+ 743us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:26:36.138016 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0810 03:26:36.138752 (+ 736us) server_negotiation.cc:299] Negotiation successful
0810 03:26:36.139203 (+ 451us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":905,"thread_start_us":562,"threads_started":1}
I20260810 03:26:36.142043 6244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:36.123784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:36281 (local address 127.0.0.1:54794)
0810 03:26:36.124783 (+ 999us) negotiation.cc:107] Waiting for socket to connect
0810 03:26:36.124844 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0810 03:26:36.125034 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:26:36.125415 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:26:36.125422 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:26:36.126759 (+ 1337us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:26:36.127167 (+ 408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:26:36.127182 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:26:36.128287 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:26:36.128293 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:26:36.130009 (+ 1716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:26:36.130033 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:26:36.131242 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:26:36.131252 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:26:36.131562 (+ 310us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:36.132265 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:26:36.132314 (+ 49us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:26:36.135052 (+ 2738us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:26:36.138830 (+ 3778us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:26:36.138843 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:26:36.138851 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:26:36.139475 (+ 624us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:26:36.139931 (+ 456us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:26:36.139934 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:26:36.139937 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:26:36.140080 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:26:36.140848 (+ 768us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:26:36.140856 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:26:36.141026 (+ 170us) client_negotiation.cc:769] Sending connection context
0810 03:26:36.141217 (+ 191us) client_negotiation.cc:240] Negotiation successful
0810 03:26:36.141580 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":735,"thread_start_us":544,"threads_started":1}
I20260810 03:26:36.142055 5969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:36.123813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54794 (local address 127.27.161.62:36281)
0810 03:26:36.123990 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0810 03:26:36.123994 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:26:36.124950 (+ 956us) server_negotiation.cc:407] Connection header received
0810 03:26:36.125164 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:26:36.125168 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:26:36.125238 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:26:36.125331 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:26:36.127355 (+ 2024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:36.128127 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:36.130173 (+ 2046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:36.130843 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:36.130949 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:36.135320 (+ 4371us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:26:36.135326 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:26:36.135330 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:26:36.135370 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:26:36.138648 (+ 3278us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:26:36.139627 (+ 979us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:26:36.139633 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:26:36.139638 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:26:36.139813 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:26:36.140217 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:26:36.140222 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:26:36.140225 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:26:36.140591 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:26:36.140750 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0810 03:26:36.141344 (+ 594us) server_negotiation.cc:299] Negotiation successful
0810 03:26:36.141646 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260810 03:26:36.143666 6239 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.161.61" port: 43223 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 03:26:36 dist-test-slave-vpvm krb5kdc[5096](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332396, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.161.60@KRBTEST.COM for oryx/127.27.161.61@KRBTEST.COM
I20260810 03:26:36.158293 6244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:36.144288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.61:43223 (local address 127.0.0.1:50784)
0810 03:26:36.144469 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0810 03:26:36.144486 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 03:26:36.144575 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:26:36.144817 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:26:36.144820 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:26:36.145262 (+ 442us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:26:36.145436 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:26:36.145445 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:26:36.146479 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:26:36.146483 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:26:36.147648 (+ 1165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:26:36.147661 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:26:36.148512 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:26:36.148518 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:26:36.148678 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:36.149342 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:26:36.149362 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:26:36.151982 (+ 2620us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:26:36.155564 (+ 3582us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:26:36.155569 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:26:36.155571 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:26:36.156136 (+ 565us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:26:36.156494 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:26:36.156498 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:26:36.156502 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:26:36.156664 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:26:36.157204 (+ 540us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:26:36.157208 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:26:36.157345 (+ 137us) client_negotiation.cc:769] Sending connection context
0810 03:26:36.157490 (+ 145us) client_negotiation.cc:240] Negotiation successful
0810 03:26:36.157846 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260810 03:26:36.158349 6146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:36.144404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50784 (local address 127.27.161.61:43223)
0810 03:26:36.144553 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0810 03:26:36.144557 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:26:36.144573 (+ 16us) server_negotiation.cc:407] Connection header received
0810 03:26:36.144656 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:26:36.144660 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:26:36.144720 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:26:36.144797 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:26:36.145546 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:36.146330 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:36.147782 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:36.148327 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:36.148425 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:36.152208 (+ 3783us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:26:36.152214 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:26:36.152217 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:26:36.152255 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:26:36.155379 (+ 3124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:26:36.156253 (+ 874us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:26:36.156258 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:26:36.156261 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:26:36.156395 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:26:36.156752 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:26:36.156756 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:26:36.156758 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:26:36.157108 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:26:36.157225 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0810 03:26:36.157639 (+ 414us) server_negotiation.cc:299] Negotiation successful
0810 03:26:36.157936 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260810 03:26:36.159395 6239 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.161.60" port: 43217 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 03:26:36 dist-test-slave-vpvm krb5kdc[5096](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332396, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.161.60@KRBTEST.COM for oryx/127.27.161.60@KRBTEST.COM
I20260810 03:26:36.172827 6242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:36.160256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46900 (local address 127.27.161.60:43217)
0810 03:26:36.160392 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0810 03:26:36.160396 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:26:36.160410 (+ 14us) server_negotiation.cc:407] Connection header received
0810 03:26:36.160449 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:26:36.160452 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:26:36.160506 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:26:36.160579 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:26:36.161386 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:36.162311 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:36.163415 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:36.163923 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:36.164024 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:36.167208 (+ 3184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:26:36.167213 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:26:36.167216 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:26:36.167246 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:26:36.169899 (+ 2653us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:26:36.170678 (+ 779us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:26:36.170684 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:26:36.170688 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:26:36.170805 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:26:36.171239 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:26:36.171245 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:26:36.171248 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:26:36.171566 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:26:36.171698 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0810 03:26:36.172063 (+ 365us) server_negotiation.cc:299] Negotiation successful
0810 03:26:36.172395 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":116,"server-negotiator.queue_time_us":45}
I20260810 03:26:36.172847 6244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:36.160074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.60:43217 (local address 127.0.0.1:46900)
0810 03:26:36.160236 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0810 03:26:36.160253 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 03:26:36.160355 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:26:36.160608 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:26:36.160611 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:26:36.161103 (+ 492us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:26:36.161269 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:26:36.161278 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:26:36.162422 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:26:36.162425 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:26:36.163297 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:26:36.163308 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:26:36.164059 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:26:36.164063 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:26:36.164222 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:36.164767 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:26:36.164785 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:26:36.167010 (+ 2225us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:26:36.170062 (+ 3052us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:26:36.170067 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:26:36.170072 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:26:36.170571 (+ 499us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:26:36.170904 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:26:36.170909 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:26:36.170913 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:26:36.171131 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:26:36.171685 (+ 554us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:26:36.171689 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:26:36.171846 (+ 157us) client_negotiation.cc:769] Sending connection context
0810 03:26:36.171965 (+ 119us) client_negotiation.cc:240] Negotiation successful
0810 03:26:36.172241 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260810 03:26:36.178705 6239 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1245ea4d6f734899a3087452b0495ee2: Bootstrap starting.
I20260810 03:26:36.182232 6239 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1245ea4d6f734899a3087452b0495ee2: Neither blocks nor log segments found. Creating new log.
I20260810 03:26:36.183684 6239 log.cc:824] T 00000000000000000000000000000000 P 1245ea4d6f734899a3087452b0495ee2: Log is configured to *not* fsync() on all Append() calls
I20260810 03:26:36.187094 6239 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1245ea4d6f734899a3087452b0495ee2: No bootstrap required, opened a new log
I20260810 03:26:36.196659 6239 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1245ea4d6f734899a3087452b0495ee2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b65187fc20346eda207dd58ee639ef2" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 36281 } } peers { permanent_uuid: "d102fadcc4b047ac9e652e1e0a4670a4" member_type: VOTER last_known_addr { host: "127.27.161.61" port: 43223 } } peers { permanent_uuid: "1245ea4d6f734899a3087452b0495ee2" member_type: VOTER last_known_addr { host: "127.27.161.60" port: 43217 } }
I20260810 03:26:36.197110 6239 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1245ea4d6f734899a3087452b0495ee2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:26:36.197248 6239 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1245ea4d6f734899a3087452b0495ee2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1245ea4d6f734899a3087452b0495ee2, State: Initialized, Role: FOLLOWER
I20260810 03:26:36.197904 6239 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1245ea4d6f734899a3087452b0495ee2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b65187fc20346eda207dd58ee639ef2" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 36281 } } peers { permanent_uuid: "d102fadcc4b047ac9e652e1e0a4670a4" member_type: VOTER last_known_addr { host: "127.27.161.61" port: 43223 } } peers { permanent_uuid: "1245ea4d6f734899a3087452b0495ee2" member_type: VOTER last_known_addr { host: "127.27.161.60" port: 43217 } }
I20260810 03:26:36.199296 6246 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1245ea4d6f734899a3087452b0495ee2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b65187fc20346eda207dd58ee639ef2" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 36281 } } peers { permanent_uuid: "d102fadcc4b047ac9e652e1e0a4670a4" member_type: VOTER last_known_addr { host: "127.27.161.61" port: 43223 } } peers { permanent_uuid: "1245ea4d6f734899a3087452b0495ee2" member_type: VOTER last_known_addr { host: "127.27.161.60" port: 43217 } } }
I20260810 03:26:36.199903 6246 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1245ea4d6f734899a3087452b0495ee2 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 03:26:36.200405 6239 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1245ea4d6f734899a3087452b0495ee2 [sys.catalog]: configured and running, proceeding with master startup.
W20260810 03:26:36.211135 6260 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1245ea4d6f734899a3087452b0495ee2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 03:26:36.211310 6260 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 03:26:36.215149 6015 master_runner.cc:428] Detected that this master 1245ea4d6f734899a3087452b0495ee2 is joining an existing cluster
I20260810 03:26:36.215283 6015 master_runner.cc:432] Initiating AddMaster RPC to add 127.27.161.60:43217
I20260810 03:26:36.241852 5969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:36.220456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54810 (local address 127.27.161.62:36281)
0810 03:26:36.220626 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0810 03:26:36.220631 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:26:36.221474 (+ 843us) server_negotiation.cc:407] Connection header received
0810 03:26:36.221657 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:26:36.221663 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:26:36.221751 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:26:36.221846 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:26:36.223119 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:36.223896 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:36.225318 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:36.225811 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:36.225926 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:36.233124 (+ 7198us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:26:36.233133 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:26:36.233139 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:26:36.233192 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:26:36.236826 (+ 3634us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:26:36.238374 (+ 1548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:26:36.238382 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:26:36.238386 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:26:36.238961 (+ 575us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:26:36.239611 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:26:36.239616 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:26:36.239620 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:26:36.240166 (+ 546us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:26:36.240332 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0810 03:26:36.241135 (+ 803us) server_negotiation.cc:299] Negotiation successful
0810 03:26:36.241428 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260810 03:26:36.243060 6269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:36.220568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:36281 (local address 127.0.0.1:54810)
0810 03:26:36.221363 (+ 795us) negotiation.cc:107] Waiting for socket to connect
0810 03:26:36.221378 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 03:26:36.221530 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:26:36.222078 (+ 548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:26:36.222083 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:26:36.222666 (+ 583us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:26:36.222967 (+ 301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:26:36.222978 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:26:36.224074 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:26:36.224079 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:26:36.225169 (+ 1090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:26:36.225184 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:26:36.230110 (+ 4926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:26:36.230123 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:26:36.230377 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:36.231319 (+ 942us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:26:36.231352 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:26:36.232846 (+ 1494us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:26:36.237068 (+ 4222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:26:36.237073 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:26:36.237078 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:26:36.238172 (+ 1094us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:26:36.239195 (+ 1023us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:26:36.239204 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:26:36.239208 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:26:36.239442 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:26:36.240440 (+ 998us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:26:36.240446 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:26:36.240734 (+ 288us) client_negotiation.cc:769] Sending connection context
0810 03:26:36.242219 (+ 1485us) client_negotiation.cc:240] Negotiation successful
0810 03:26:36.242622 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":651,"mutex_wait_us":1,"thread_start_us":511,"threads_started":1}
I20260810 03:26:36.243920 6270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:36.221820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.61:43223 (local address 127.0.0.1:50788)
0810 03:26:36.226541 (+ 4721us) negotiation.cc:107] Waiting for socket to connect
0810 03:26:36.226562 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0810 03:26:36.226856 (+ 294us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:26:36.227508 (+ 652us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:26:36.227514 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:26:36.228347 (+ 833us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:26:36.228815 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:26:36.228830 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:26:36.230281 (+ 1451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:26:36.230287 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:26:36.232057 (+ 1770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:26:36.232072 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:26:36.233328 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:26:36.233334 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:26:36.234091 (+ 757us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:36.234982 (+ 891us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:26:36.235011 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:26:36.236357 (+ 1346us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:26:36.240630 (+ 4273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:26:36.240639 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:26:36.240642 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:26:36.241199 (+ 557us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:26:36.241660 (+ 461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:26:36.241665 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:26:36.241669 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:26:36.241842 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:26:36.242577 (+ 735us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:26:36.242583 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:26:36.242887 (+ 304us) client_negotiation.cc:769] Sending connection context
0810 03:26:36.243051 (+ 164us) client_negotiation.cc:240] Negotiation successful
0810 03:26:36.243414 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4548,"mutex_wait_us":47,"thread_start_us":4330,"threads_started":1}
I20260810 03:26:36.245201 6146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:36.221880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50788 (local address 127.27.161.61:43223)
0810 03:26:36.222087 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0810 03:26:36.222093 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:26:36.226722 (+ 4629us) server_negotiation.cc:407] Connection header received
0810 03:26:36.227088 (+ 366us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:26:36.227094 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:26:36.227195 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:26:36.227301 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:26:36.229004 (+ 1703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:36.230090 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:36.232294 (+ 2204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:36.232938 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:36.233054 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:36.236589 (+ 3535us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:26:36.236596 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:26:36.236600 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:26:36.236645 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:26:36.240460 (+ 3815us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:26:36.241355 (+ 895us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:26:36.241360 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:26:36.241363 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:26:36.241521 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:26:36.242010 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:26:36.242016 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:26:36.242020 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:26:36.242367 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:26:36.242511 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0810 03:26:36.244596 (+ 2085us) server_negotiation.cc:299] Negotiation successful
0810 03:26:36.244856 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260810 03:26:36.246038 6242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:36.226723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46916 (local address 127.27.161.60:43217)
0810 03:26:36.226918 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0810 03:26:36.226924 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:26:36.227614 (+ 690us) server_negotiation.cc:407] Connection header received
0810 03:26:36.227792 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:26:36.227798 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:26:36.227863 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:26:36.227958 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:26:36.229387 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:36.230417 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:36.232701 (+ 2284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:36.233646 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:36.233773 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:36.236512 (+ 2739us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:26:36.236519 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:26:36.236525 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:26:36.236566 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:26:36.240882 (+ 4316us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:26:36.241701 (+ 819us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:26:36.241707 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:26:36.241710 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:26:36.241886 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:26:36.242360 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:26:36.242365 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:26:36.242369 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:26:36.244261 (+ 1892us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:26:36.244429 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0810 03:26:36.245159 (+ 730us) server_negotiation.cc:299] Negotiation successful
0810 03:26:36.245636 (+ 477us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1440,"server-negotiator.queue_time_us":62}
I20260810 03:26:36.247272 6273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:36.226631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.60:43217 (local address 127.0.0.1:46916)
0810 03:26:36.227511 (+ 880us) negotiation.cc:107] Waiting for socket to connect
0810 03:26:36.227527 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 03:26:36.227660 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:26:36.228001 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:26:36.228006 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:26:36.228878 (+ 872us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:26:36.229256 (+ 378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:26:36.229266 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:26:36.230718 (+ 1452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:26:36.230724 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:26:36.232522 (+ 1798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:26:36.232540 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:26:36.233871 (+ 1331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:26:36.233879 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:26:36.234175 (+ 296us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:36.234865 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:26:36.234894 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:26:36.236277 (+ 1383us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:26:36.241026 (+ 4749us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:26:36.241031 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:26:36.241034 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:26:36.241564 (+ 530us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:26:36.242078 (+ 514us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:26:36.242083 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:26:36.242086 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:26:36.242243 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:26:36.244427 (+ 2184us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:26:36.244433 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:26:36.244814 (+ 381us) client_negotiation.cc:769] Sending connection context
0810 03:26:36.246408 (+ 1594us) client_negotiation.cc:240] Negotiation successful
0810 03:26:36.246701 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":759,"mutex_wait_us":22,"thread_start_us":574,"threads_started":1}
I20260810 03:26:36.247514 28292 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iBH5/build/asan/bin/kudu pid 5247
Aug 10 03:26:36 dist-test-slave-vpvm krb5kdc[5096](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332392, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.27.161.1@KRBTEST.COM
Aug 10 03:26:36 dist-test-slave-vpvm krb5kdc[5096](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332392, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.161.62@KRBTEST.COM for oryx/127.27.161.60@KRBTEST.COM
I20260810 03:26:36.273406 6004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:36.254357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.60:43217 (local address 127.0.0.1:46924)
0810 03:26:36.254676 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0810 03:26:36.254697 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0810 03:26:36.254819 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:26:36.258622 (+ 3803us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:26:36.258629 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:26:36.259196 (+ 567us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:26:36.259431 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:26:36.259446 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:26:36.260689 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:26:36.260696 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:26:36.261642 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:26:36.261656 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:26:36.262613 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:26:36.262620 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:26:36.262822 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:36.263498 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:26:36.263528 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:26:36.266241 (+ 2713us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:26:36.270162 (+ 3921us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:26:36.270168 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:26:36.270173 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:26:36.270676 (+ 503us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:26:36.271173 (+ 497us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:26:36.271179 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:26:36.271183 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:26:36.271401 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:26:36.272106 (+ 705us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:26:36.272111 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:26:36.272275 (+ 164us) client_negotiation.cc:769] Sending connection context
0810 03:26:36.272412 (+ 137us) client_negotiation.cc:240] Negotiation successful
0810 03:26:36.272779 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176}
I20260810 03:26:36.273386 6242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:36.257408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46924 (local address 127.27.161.60:43217)
0810 03:26:36.257889 (+ 481us) server_negotiation.cc:206] Beginning negotiation
0810 03:26:36.257895 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:26:36.257916 (+ 21us) server_negotiation.cc:407] Connection header received
0810 03:26:36.258080 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:26:36.258086 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:26:36.258231 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:26:36.258342 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:26:36.259618 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:36.260540 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:36.261804 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:36.262422 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:36.262590 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:36.266512 (+ 3922us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:26:36.266521 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:26:36.266526 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:26:36.266576 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:26:36.269945 (+ 3369us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:26:36.270814 (+ 869us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:26:36.270821 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:26:36.270824 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:26:36.271011 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:26:36.271539 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:26:36.271544 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:26:36.271547 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:26:36.271970 (+ 423us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:26:36.272109 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0810 03:26:36.272539 (+ 430us) server_negotiation.cc:299] Negotiation successful
0810 03:26:36.272781 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":304}
I20260810 03:26:36.276463 5158 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.27.161.60:43217: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "1245ea4d6f734899a3087452b0495ee2"
member_type: NON_VOTER
last_known_addr {
host: "127.27.161.60"
port: 43217
}
attrs {
promote: true
}
}
dest_uuid: "4b65187fc20346eda207dd58ee639ef2"
cas_config_opid_index: 5
I20260810 03:26:36.276922 6275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:36.249387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51552 (local address 127.27.161.1:36623)
0810 03:26:36.250472 (+ 1085us) server_negotiation.cc:206] Beginning negotiation
0810 03:26:36.250484 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 03:26:36.250551 (+ 67us) server_negotiation.cc:407] Connection header received
0810 03:26:36.250695 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:26:36.250708 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:26:36.251051 (+ 343us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:26:36.251240 (+ 189us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:26:36.254789 (+ 3549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:36.256037 (+ 1248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:36.258069 (+ 2032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:36.259056 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:36.259215 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:36.265420 (+ 6205us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:26:36.265445 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:26:36.265464 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:26:36.265523 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:26:36.269981 (+ 4458us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:26:36.271619 (+ 1638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:26:36.271630 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:26:36.271633 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:26:36.271805 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:26:36.272349 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:26:36.272355 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:26:36.272359 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:26:36.272856 (+ 497us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:26:36.273373 (+ 517us) server_negotiation.cc:1035] Waiting for connection context
0810 03:26:36.275586 (+ 2213us) server_negotiation.cc:299] Negotiation successful
0810 03:26:36.276114 (+ 528us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":788,"thread_start_us":754,"threads_started":1}
I20260810 03:26:36.277987 5158 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4b65187fc20346eda207dd58ee639ef2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4b65187fc20346eda207dd58ee639ef2" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 36281 } } peers { permanent_uuid: "d102fadcc4b047ac9e652e1e0a4670a4" member_type: NON_VOTER last_known_addr { host: "127.27.161.61" port: 43223 } attrs { promote: true } } peers { permanent_uuid: "1245ea4d6f734899a3087452b0495ee2" member_type: NON_VOTER last_known_addr { host: "127.27.161.60" port: 43217 } attrs { promote: true } }
I20260810 03:26:36.280815 5436 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:26:36.280817 6008 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 4b65187fc20346eda207dd58ee639ef2 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 1245ea4d6f734899a3087452b0495ee2 (127.27.161.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4b65187fc20346eda207dd58ee639ef2" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 36281 } } peers { permanent_uuid: "d102fadcc4b047ac9e652e1e0a4670a4" member_type: NON_VOTER last_known_addr { host: "127.27.161.61" port: 43223 } attrs { promote: true } } peers { permanent_uuid: "1245ea4d6f734899a3087452b0495ee2" member_type: NON_VOTER last_known_addr { host: "127.27.161.60" port: 43217 } attrs { promote: true } } }
I20260810 03:26:36.282076 6083 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d102fadcc4b047ac9e652e1e0a4670a4 [term 1 LEARNER]: Refusing update from remote peer 4b65187fc20346eda207dd58ee639ef2: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260810 03:26:36.285622 6204 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1245ea4d6f734899a3087452b0495ee2 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:26:36.287426 6008 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.27.161.60:43217
I20260810 03:26:36.298815 6278 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4b65187fc20346eda207dd58ee639ef2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d102fadcc4b047ac9e652e1e0a4670a4" member_type: NON_VOTER last_known_addr { host: "127.27.161.61" port: 43223 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260810 03:26:36.303335 6278 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b65187fc20346eda207dd58ee639ef2 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "4b65187fc20346eda207dd58ee639ef2" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4b65187fc20346eda207dd58ee639ef2" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 36281 } } peers { permanent_uuid: "d102fadcc4b047ac9e652e1e0a4670a4" member_type: NON_VOTER last_known_addr { host: "127.27.161.61" port: 43223 } attrs { promote: true } } peers { permanent_uuid: "1245ea4d6f734899a3087452b0495ee2" member_type: NON_VOTER last_known_addr { host: "127.27.161.60" port: 43217 } attrs { promote: true } } }
I20260810 03:26:36.304116 6278 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b65187fc20346eda207dd58ee639ef2 [sys.catalog]: This master's current role is: LEADER
I20260810 03:26:36.313179 6083 raft_consensus.cc:2932] T 00000000000000000000000000000000 P d102fadcc4b047ac9e652e1e0a4670a4 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 1245ea4d6f734899a3087452b0495ee2 (127.27.161.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4b65187fc20346eda207dd58ee639ef2" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 36281 } } peers { permanent_uuid: "d102fadcc4b047ac9e652e1e0a4670a4" member_type: NON_VOTER last_known_addr { host: "127.27.161.61" port: 43223 } attrs { promote: true } } peers { permanent_uuid: "1245ea4d6f734899a3087452b0495ee2" member_type: NON_VOTER last_known_addr { host: "127.27.161.60" port: 43217 } attrs { promote: true } } }
I20260810 03:26:36.315099 6204 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1245ea4d6f734899a3087452b0495ee2 [term 1 FOLLOWER]: Refusing update from remote peer 4b65187fc20346eda207dd58ee639ef2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260810 03:26:36.316156 6015 master.cc:537] Master@127.27.161.60:43217 shutting down...
I20260810 03:26:36.316743 6246 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1245ea4d6f734899a3087452b0495ee2 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 03:26:36.317278 6246 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1245ea4d6f734899a3087452b0495ee2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b65187fc20346eda207dd58ee639ef2" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 36281 } } peers { permanent_uuid: "d102fadcc4b047ac9e652e1e0a4670a4" member_type: VOTER last_known_addr { host: "127.27.161.61" port: 43223 } } peers { permanent_uuid: "1245ea4d6f734899a3087452b0495ee2" member_type: VOTER last_known_addr { host: "127.27.161.60" port: 43217 } }
I20260810 03:26:36.317276 6278 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4b65187fc20346eda207dd58ee639ef2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1245ea4d6f734899a3087452b0495ee2" member_type: NON_VOTER last_known_addr { host: "127.27.161.60" port: 43217 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.001s
I20260810 03:26:36.322883 5173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1245ea4d6f734899a3087452b0495ee2" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b65187fc20346eda207dd58ee639ef2" is_pre_election: true
W20260810 03:26:36.326108 6175 tablet.cc:2369] T 00000000000000000000000000000000 P 1245ea4d6f734899a3087452b0495ee2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 03:26:36.327304 6081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1245ea4d6f734899a3087452b0495ee2" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d102fadcc4b047ac9e652e1e0a4670a4" is_pre_election: true
I20260810 03:26:36.329756 6167 leader_election.cc:302] T 00000000000000000000000000000000 P 1245ea4d6f734899a3087452b0495ee2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1245ea4d6f734899a3087452b0495ee2; no voters: 4b65187fc20346eda207dd58ee639ef2, d102fadcc4b047ac9e652e1e0a4670a4
W20260810 03:26:36.331799 5135 consensus_peers.cc:487] T 00000000000000000000000000000000 P 4b65187fc20346eda207dd58ee639ef2 -> Peer 1245ea4d6f734899a3087452b0495ee2 (127.27.161.60:43217): Couldn't send request to peer 1245ea4d6f734899a3087452b0495ee2. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260810 03:26:36.332888 6246 leader_election.cc:288] T 00000000000000000000000000000000 P 1245ea4d6f734899a3087452b0495ee2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4b65187fc20346eda207dd58ee639ef2 (127.27.161.62:36281), d102fadcc4b047ac9e652e1e0a4670a4 (127.27.161.61:43223)
I20260810 03:26:36.332988 6289 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 1245ea4d6f734899a3087452b0495ee2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 03:26:36.336678 6125 sys_catalog.cc:455] T 00000000000000000000000000000000 P d102fadcc4b047ac9e652e1e0a4670a4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4b65187fc20346eda207dd58ee639ef2" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4b65187fc20346eda207dd58ee639ef2" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 36281 } } peers { permanent_uuid: "d102fadcc4b047ac9e652e1e0a4670a4" member_type: NON_VOTER last_known_addr { host: "127.27.161.61" port: 43223 } attrs { promote: true } } peers { permanent_uuid: "1245ea4d6f734899a3087452b0495ee2" member_type: NON_VOTER last_known_addr { host: "127.27.161.60" port: 43217 } attrs { promote: true } } }
I20260810 03:26:36.337262 6125 sys_catalog.cc:458] T 00000000000000000000000000000000 P d102fadcc4b047ac9e652e1e0a4670a4 [sys.catalog]: This master's current role is: LEARNER
I20260810 03:26:36.339236 6278 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 4b65187fc20346eda207dd58ee639ef2: attempting to promote NON_VOTER d102fadcc4b047ac9e652e1e0a4670a4 to VOTER
I20260810 03:26:36.340961 6278 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4b65187fc20346eda207dd58ee639ef2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4b65187fc20346eda207dd58ee639ef2" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 36281 } } peers { permanent_uuid: "d102fadcc4b047ac9e652e1e0a4670a4" member_type: VOTER last_known_addr { host: "127.27.161.61" port: 43223 } attrs { promote: false } } peers { permanent_uuid: "1245ea4d6f734899a3087452b0495ee2" member_type: NON_VOTER last_known_addr { host: "127.27.161.60" port: 43217 } attrs { promote: true } }
I20260810 03:26:36.345857 6083 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d102fadcc4b047ac9e652e1e0a4670a4 [term 1 LEARNER]: Refusing update from remote peer 4b65187fc20346eda207dd58ee639ef2: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20260810 03:26:36.346617 5135 consensus_peers.cc:487] T 00000000000000000000000000000000 P 4b65187fc20346eda207dd58ee639ef2 -> Peer 1245ea4d6f734899a3087452b0495ee2 (127.27.161.60:43217): Couldn't send request to peer 1245ea4d6f734899a3087452b0495ee2. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260810 03:26:36.347268 6276 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4b65187fc20346eda207dd58ee639ef2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d102fadcc4b047ac9e652e1e0a4670a4" member_type: VOTER last_known_addr { host: "127.27.161.61" port: 43223 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260810 03:26:36.351042 6015 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 1245ea4d6f734899a3087452b0495ee2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 03:26:36.351891 6015 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 1245ea4d6f734899a3087452b0495ee2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:26:36.352555 6015 tablet_replica.cc:330] T 00000000000000000000000000000000 P 1245ea4d6f734899a3087452b0495ee2: stopping tablet replica
I20260810 03:26:36.353492 6276 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 4b65187fc20346eda207dd58ee639ef2 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, d102fadcc4b047ac9e652e1e0a4670a4 (127.27.161.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4b65187fc20346eda207dd58ee639ef2" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 36281 } } peers { permanent_uuid: "d102fadcc4b047ac9e652e1e0a4670a4" member_type: VOTER last_known_addr { host: "127.27.161.61" port: 43223 } attrs { promote: false } } peers { permanent_uuid: "1245ea4d6f734899a3087452b0495ee2" member_type: NON_VOTER last_known_addr { host: "127.27.161.60" port: 43217 } attrs { promote: true } } }
I20260810 03:26:36.357648 6083 raft_consensus.cc:2932] T 00000000000000000000000000000000 P d102fadcc4b047ac9e652e1e0a4670a4 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, d102fadcc4b047ac9e652e1e0a4670a4 (127.27.161.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4b65187fc20346eda207dd58ee639ef2" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 36281 } } peers { permanent_uuid: "d102fadcc4b047ac9e652e1e0a4670a4" member_type: VOTER last_known_addr { host: "127.27.161.61" port: 43223 } attrs { promote: false } } peers { permanent_uuid: "1245ea4d6f734899a3087452b0495ee2" member_type: NON_VOTER last_known_addr { host: "127.27.161.60" port: 43217 } attrs { promote: true } } }
I20260810 03:26:36.362607 6278 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b65187fc20346eda207dd58ee639ef2 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "4b65187fc20346eda207dd58ee639ef2" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4b65187fc20346eda207dd58ee639ef2" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 36281 } } peers { permanent_uuid: "d102fadcc4b047ac9e652e1e0a4670a4" member_type: VOTER last_known_addr { host: "127.27.161.61" port: 43223 } attrs { promote: false } } peers { permanent_uuid: "1245ea4d6f734899a3087452b0495ee2" member_type: NON_VOTER last_known_addr { host: "127.27.161.60" port: 43217 } attrs { promote: true } } }
I20260810 03:26:36.363368 6278 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b65187fc20346eda207dd58ee639ef2 [sys.catalog]: This master's current role is: LEADER
I20260810 03:26:36.364120 6278 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b65187fc20346eda207dd58ee639ef2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4b65187fc20346eda207dd58ee639ef2" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4b65187fc20346eda207dd58ee639ef2" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 36281 } } peers { permanent_uuid: "d102fadcc4b047ac9e652e1e0a4670a4" member_type: VOTER last_known_addr { host: "127.27.161.61" port: 43223 } attrs { promote: false } } peers { permanent_uuid: "1245ea4d6f734899a3087452b0495ee2" member_type: NON_VOTER last_known_addr { host: "127.27.161.60" port: 43217 } attrs { promote: true } } }
I20260810 03:26:36.364763 6278 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b65187fc20346eda207dd58ee639ef2 [sys.catalog]: This master's current role is: LEADER
I20260810 03:26:36.373112 6125 sys_catalog.cc:455] T 00000000000000000000000000000000 P d102fadcc4b047ac9e652e1e0a4670a4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4b65187fc20346eda207dd58ee639ef2" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4b65187fc20346eda207dd58ee639ef2" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 36281 } } peers { permanent_uuid: "d102fadcc4b047ac9e652e1e0a4670a4" member_type: VOTER last_known_addr { host: "127.27.161.61" port: 43223 } attrs { promote: false } } peers { permanent_uuid: "1245ea4d6f734899a3087452b0495ee2" member_type: NON_VOTER last_known_addr { host: "127.27.161.60" port: 43217 } attrs { promote: true } } }
I20260810 03:26:36.373777 6125 sys_catalog.cc:458] T 00000000000000000000000000000000 P d102fadcc4b047ac9e652e1e0a4670a4 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 03:26:36.385038 6015 master.cc:552] Master@127.27.161.60:43217 shutdown complete.
I20260810 03:26:36.385181 6015 master_runner.cc:305] Clearing existing system tablet
I20260810 03:26:36.385784 6015 fs_manager.cc:359] Metadata directory not provided
I20260810 03:26:36.385937 6015 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:26:36.391111 6015 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.004s
I20260810 03:26:36.393289 6292 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:26:36.393787 6015 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786332336319544-28292-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:26:36.394251 6015 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260810 03:26:36.394418 6015 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786332336319544-28292-0/minicluster-data/master-2/data,/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786332336319544-28292-0/minicluster-data/master-2/wal
uuid: "1245ea4d6f734899a3087452b0495ee2"
format_stamp: "Formatted at 2026-08-10 03:26:36 on dist-test-slave-vpvm"
server_key: "73691ae5721db7f7d304a0e61ed586ba"
server_key_iv: "6ebc903fbb147ef98be58a062654bcf0"
server_key_version: "encryptionkey@0"
I20260810 03:26:36.399629 6015 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 1245ea4d6f734899a3087452b0495ee2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:26:36.409004 6015 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 1245ea4d6f734899a3087452b0495ee2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260810 03:26:36.409349 6015 log.cc:1196] T 00000000000000000000000000000000 P 1245ea4d6f734899a3087452b0495ee2: Deleting WAL directory at /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786332336319544-28292-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260810 03:26:36.410140 6015 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 1245ea4d6f734899a3087452b0495ee2: Deleting consensus metadata
I20260810 03:26:36.410581 6015 master_runner.cc:315] Copying system tablet from 127.27.161.62:36281
I20260810 03:26:36.414911 6015 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 1245ea4d6f734899a3087452b0495ee2: tablet copy: Beginning tablet copy session from remote peer at address 127.27.161.62:36281
I20260810 03:26:36.437619 5969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:36.416603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54826 (local address 127.27.161.62:36281)
0810 03:26:36.417095 (+ 492us) server_negotiation.cc:206] Beginning negotiation
0810 03:26:36.417112 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0810 03:26:36.418224 (+ 1112us) server_negotiation.cc:407] Connection header received
0810 03:26:36.418643 (+ 419us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:26:36.418649 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:26:36.418785 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:26:36.419013 (+ 228us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:26:36.420598 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:36.421758 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:36.423915 (+ 2157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:36.424665 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:36.424901 (+ 236us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:36.428310 (+ 3409us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:26:36.428318 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:26:36.428321 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:26:36.428375 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:26:36.432989 (+ 4614us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:26:36.434038 (+ 1049us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:26:36.434048 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:26:36.434051 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:26:36.434405 (+ 354us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:26:36.435021 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:26:36.435027 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:26:36.435030 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:26:36.435614 (+ 584us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:26:36.435922 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0810 03:26:36.436519 (+ 597us) server_negotiation.cc:299] Negotiation successful
0810 03:26:36.436877 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":259}
I20260810 03:26:36.438192 6302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:36.416490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:36281 (local address 127.0.0.1:54826)
0810 03:26:36.417992 (+ 1502us) negotiation.cc:107] Waiting for socket to connect
0810 03:26:36.418087 (+ 95us) client_negotiation.cc:174] Beginning negotiation
0810 03:26:36.418365 (+ 278us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:26:36.419119 (+ 754us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:26:36.419136 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:26:36.419938 (+ 802us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:26:36.420354 (+ 416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:26:36.420376 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:26:36.422049 (+ 1673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:26:36.422058 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:26:36.423692 (+ 1634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:26:36.423718 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:26:36.424945 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:26:36.424961 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:26:36.425363 (+ 402us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:36.426611 (+ 1248us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:26:36.426674 (+ 63us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:26:36.427944 (+ 1270us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:26:36.433214 (+ 5270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:26:36.433222 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:26:36.433226 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:26:36.433805 (+ 579us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:26:36.434614 (+ 809us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:26:36.434618 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:26:36.434621 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:26:36.434858 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:26:36.435801 (+ 943us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:26:36.435809 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:26:36.436128 (+ 319us) client_negotiation.cc:769] Sending connection context
0810 03:26:36.436315 (+ 187us) client_negotiation.cc:240] Negotiation successful
0810 03:26:36.436814 (+ 499us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1133,"thread_start_us":863,"threads_started":1}
I20260810 03:26:36.439107 5190 tablet_copy_service.cc:140] P 4b65187fc20346eda207dd58ee639ef2: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 1245ea4d6f734899a3087452b0495ee2 ({username='oryx', principal='oryx/127.27.161.60@KRBTEST.COM'} at 127.0.0.1:54826)
I20260810 03:26:36.439348 5190 tablet_copy_service.cc:161] P 4b65187fc20346eda207dd58ee639ef2: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 1245ea4d6f734899a3087452b0495ee2 at {username='oryx', principal='oryx/127.27.161.60@KRBTEST.COM'} at 127.0.0.1:54826: session id = 1245ea4d6f734899a3087452b0495ee2-00000000000000000000000000000000
I20260810 03:26:36.442927 5190 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 4b65187fc20346eda207dd58ee639ef2: Tablet Copy: opened 0 blocks and 1 log segments
I20260810 03:26:36.445389 6015 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260810 03:26:36.453939 6015 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 1245ea4d6f734899a3087452b0495ee2: tablet copy: Starting download of 0 data blocks...
I20260810 03:26:36.454591 6015 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 1245ea4d6f734899a3087452b0495ee2: tablet copy: Starting download of 1 WAL segments...
I20260810 03:26:36.458556 6015 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 1245ea4d6f734899a3087452b0495ee2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260810 03:26:36.464244 5190 tablet_copy_service.cc:342] P 4b65187fc20346eda207dd58ee639ef2: Request end of tablet copy session 1245ea4d6f734899a3087452b0495ee2-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.27.161.60@KRBTEST.COM'} at 127.0.0.1:54826
I20260810 03:26:36.464488 5190 tablet_copy_service.cc:433] P 4b65187fc20346eda207dd58ee639ef2: ending tablet copy session 1245ea4d6f734899a3087452b0495ee2-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 1245ea4d6f734899a3087452b0495ee2
I20260810 03:26:36.485703 6015 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:26:36.491096 6305 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:26:36.502847 6306 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:26:36.506845 6015 server_base.cc:1034] running on GCE node
W20260810 03:26:36.507540 6308 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:26:36.508538 6015 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:26:36.511121 6015 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:26:36.512415 6015 hybrid_clock.cc:648] HybridClock initialized: now 1786332396512339 us; error 84 us; skew 500 ppm
I20260810 03:26:36.512820 6015 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:26:36 dist-test-slave-vpvm krb5kdc[5096](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332396, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.161.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:26:36.516328 6015 init.cc:377] Logged in from keytab as oryx/127.27.161.60@KRBTEST.COM (short username oryx)
I20260810 03:26:36.516418 28292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iBH5/build/asan/bin/kudu with pid 5247
I20260810 03:26:36.549155 6015 webserver.cc:460] Webserver started at http://127.27.161.60:44135/ using document root <none> and password file <none>
I20260810 03:26:36.549724 6015 fs_manager.cc:359] Metadata directory not provided
I20260810 03:26:36.549830 6015 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:26:36.554314 6015 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260810 03:26:36.558399 6319 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:26:36.559065 6015 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260810 03:26:36.559402 6015 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786332336319544-28292-0/minicluster-data/master-2/data,/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786332336319544-28292-0/minicluster-data/master-2/wal
uuid: "1245ea4d6f734899a3087452b0495ee2"
format_stamp: "Formatted at 2026-08-10 03:26:36 on dist-test-slave-vpvm"
server_key: "73691ae5721db7f7d304a0e61ed586ba"
server_key_iv: "6ebc903fbb147ef98be58a062654bcf0"
server_key_version: "encryptionkey@0"
I20260810 03:26:36.559944 6015 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786332336319544-28292-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786332336319544-28292-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786332336319544-28292-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:26:36.584900 6015 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:26:36.589166 28292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iBH5/build/asan/bin/kudu
/tmp/dist-test-task24iBH5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786332336319544-28292-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786332336319544-28292-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786332336319544-28292-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786332336319544-28292-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.161.1:36623
--local_ip_for_outbound_sockets=127.27.161.1
--tserver_master_addrs=127.27.161.62:36281
--webserver_port=39437
--webserver_interface=127.27.161.1
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786332336319544-28292-0/krb5kdc/oryx_127.27.161.1.keytab
--principal=oryx/127.27.161.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.161.20:43095
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.27.161.62:36281,127.27.161.61:43223
--tserver_master_addrs=127.27.161.62:36281,127.27.161.61:43223,127.27.161.60:43217 with env {KRB5CCNAME=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786332336319544-28292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786332336319544-28292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786332336319544-28292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 03:26:36.591338 6015 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:26:36.650352 6015 rpc_server.cc:307] RPC server started. Bound to: 127.27.161.60:43217
I20260810 03:26:36.650635 6400 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.161.60:43217 every 8 connection(s)
I20260810 03:26:36.652021 6015 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786332336319544-28292-0/minicluster-data/master-2/data/info.pb
I20260810 03:26:36.662087 6402 sys_catalog.cc:263] Verifying existing consensus state
I20260810 03:26:36.669689 6402 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1245ea4d6f734899a3087452b0495ee2: Bootstrap starting.
I20260810 03:26:36.710678 6402 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1245ea4d6f734899a3087452b0495ee2: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 03:26:36.712126 6402 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1245ea4d6f734899a3087452b0495ee2: Bootstrap complete.
I20260810 03:26:36.714474 6402 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1245ea4d6f734899a3087452b0495ee2 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4b65187fc20346eda207dd58ee639ef2" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 36281 } } peers { permanent_uuid: "d102fadcc4b047ac9e652e1e0a4670a4" member_type: VOTER last_known_addr { host: "127.27.161.61" port: 43223 } attrs { promote: false } } peers { permanent_uuid: "1245ea4d6f734899a3087452b0495ee2" member_type: NON_VOTER last_known_addr { host: "127.27.161.60" port: 43217 } attrs { promote: true } }
I20260810 03:26:36.715390 6402 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1245ea4d6f734899a3087452b0495ee2 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1245ea4d6f734899a3087452b0495ee2, State: Initialized, Role: LEARNER
I20260810 03:26:36.716116 6402 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1245ea4d6f734899a3087452b0495ee2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4b65187fc20346eda207dd58ee639ef2" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 36281 } } peers { permanent_uuid: "d102fadcc4b047ac9e652e1e0a4670a4" member_type: VOTER last_known_addr { host: "127.27.161.61" port: 43223 } attrs { promote: false } } peers { permanent_uuid: "1245ea4d6f734899a3087452b0495ee2" member_type: NON_VOTER last_known_addr { host: "127.27.161.60" port: 43217 } attrs { promote: true } }
I20260810 03:26:36.718557 6405 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1245ea4d6f734899a3087452b0495ee2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4b65187fc20346eda207dd58ee639ef2" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 36281 } } peers { permanent_uuid: "d102fadcc4b047ac9e652e1e0a4670a4" member_type: VOTER last_known_addr { host: "127.27.161.61" port: 43223 } attrs { promote: false } } peers { permanent_uuid: "1245ea4d6f734899a3087452b0495ee2" member_type: NON_VOTER last_known_addr { host: "127.27.161.60" port: 43217 } attrs { promote: true } } }
I20260810 03:26:36.719296 6405 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1245ea4d6f734899a3087452b0495ee2 [sys.catalog]: This master's current role is: LEARNER
I20260810 03:26:36.719868 6402 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1245ea4d6f734899a3087452b0495ee2 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:26:36.742022 6420 catalog_manager.cc:1259] Loaded cluster ID: 75ff020955394fb1943d772cebcad889
I20260810 03:26:36.742357 6420 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 1245ea4d6f734899a3087452b0495ee2: loading cluster ID for follower catalog manager: success
I20260810 03:26:36.749512 6420 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 1245ea4d6f734899a3087452b0495ee2: acquiring CA information for follower catalog manager: success
I20260810 03:26:36.755247 6420 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 1245ea4d6f734899a3087452b0495ee2: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 03:26:36.967150 6423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:36.940080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.60:43217 (local address 127.0.0.1:46932)
0810 03:26:36.953111 (+ 13031us) negotiation.cc:107] Waiting for socket to connect
0810 03:26:36.953158 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0810 03:26:36.953446 (+ 288us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:26:36.955972 (+ 2526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:26:36.955978 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:26:36.956014 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:26:36.956272 (+ 258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:26:36.956290 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:26:36.959395 (+ 3105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:26:36.959403 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:26:36.962748 (+ 3345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:26:36.962788 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:26:36.965651 (+ 2863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:26:36.965666 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:26:36.966070 (+ 404us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:36.966156 (+ 86us) client_negotiation.cc:769] Sending connection context
0810 03:26:36.966340 (+ 184us) client_negotiation.cc:240] Negotiation successful
0810 03:26:36.966606 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12756,"thread_start_us":13434,"threads_started":1}
I20260810 03:26:36.967216 6428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:36.940580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46932 (local address 127.27.161.60:43217)
0810 03:26:36.954945 (+ 14365us) server_negotiation.cc:206] Beginning negotiation
0810 03:26:36.954974 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0810 03:26:36.955005 (+ 31us) server_negotiation.cc:407] Connection header received
0810 03:26:36.955235 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:26:36.955244 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:26:36.955506 (+ 262us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:26:36.955759 (+ 253us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:26:36.956552 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:36.959141 (+ 2589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:36.963089 (+ 3948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:36.965293 (+ 2204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:36.965576 (+ 283us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:36.966101 (+ 525us) server_negotiation.cc:1035] Waiting for connection context
0810 03:26:36.966395 (+ 294us) server_negotiation.cc:299] Negotiation successful
0810 03:26:36.966622 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13954,"thread_start_us":13662,"threads_started":1}
I20260810 03:26:36.969683 6359 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 1245ea4d6f734899a3087452b0495ee2 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260810 03:26:36.970856 6405 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1245ea4d6f734899a3087452b0495ee2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4b65187fc20346eda207dd58ee639ef2. Latest consensus state: current_term: 1 leader_uuid: "4b65187fc20346eda207dd58ee639ef2" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4b65187fc20346eda207dd58ee639ef2" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 36281 } } peers { permanent_uuid: "d102fadcc4b047ac9e652e1e0a4670a4" member_type: VOTER last_known_addr { host: "127.27.161.61" port: 43223 } attrs { promote: false } } peers { permanent_uuid: "1245ea4d6f734899a3087452b0495ee2" member_type: NON_VOTER last_known_addr { host: "127.27.161.60" port: 43217 } attrs { promote: true } } }
I20260810 03:26:36.971470 6405 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1245ea4d6f734899a3087452b0495ee2 [sys.catalog]: This master's current role is: LEARNER
I20260810 03:26:37.351590 6421 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b65187fc20346eda207dd58ee639ef2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4b65187fc20346eda207dd58ee639ef2" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4b65187fc20346eda207dd58ee639ef2" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 36281 } } peers { permanent_uuid: "d102fadcc4b047ac9e652e1e0a4670a4" member_type: VOTER last_known_addr { host: "127.27.161.61" port: 43223 } attrs { promote: false } } peers { permanent_uuid: "1245ea4d6f734899a3087452b0495ee2" member_type: NON_VOTER last_known_addr { host: "127.27.161.60" port: 43217 } attrs { promote: true } } }
I20260810 03:26:37.353549 6421 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b65187fc20346eda207dd58ee639ef2 [sys.catalog]: This master's current role is: LEADER
I20260810 03:26:37.356546 6431 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 4b65187fc20346eda207dd58ee639ef2: attempting to promote NON_VOTER 1245ea4d6f734899a3087452b0495ee2 to VOTER
I20260810 03:26:37.360349 6431 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4b65187fc20346eda207dd58ee639ef2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4b65187fc20346eda207dd58ee639ef2" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 36281 } } peers { permanent_uuid: "d102fadcc4b047ac9e652e1e0a4670a4" member_type: VOTER last_known_addr { host: "127.27.161.61" port: 43223 } attrs { promote: false } } peers { permanent_uuid: "1245ea4d6f734899a3087452b0495ee2" member_type: VOTER last_known_addr { host: "127.27.161.60" port: 43217 } attrs { promote: false } }
I20260810 03:26:37.370270 6083 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d102fadcc4b047ac9e652e1e0a4670a4 [term 1 FOLLOWER]: Refusing update from remote peer 4b65187fc20346eda207dd58ee639ef2: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260810 03:26:37.372224 6359 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1245ea4d6f734899a3087452b0495ee2 [term 1 LEARNER]: Refusing update from remote peer 4b65187fc20346eda207dd58ee639ef2: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260810 03:26:37.373483 6421 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4b65187fc20346eda207dd58ee639ef2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d102fadcc4b047ac9e652e1e0a4670a4" member_type: VOTER last_known_addr { host: "127.27.161.61" port: 43223 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260810 03:26:37.375413 6429 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4b65187fc20346eda207dd58ee639ef2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1245ea4d6f734899a3087452b0495ee2" member_type: VOTER last_known_addr { host: "127.27.161.60" port: 43217 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260810 03:26:37.388505 6421 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 4b65187fc20346eda207dd58ee639ef2 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 1245ea4d6f734899a3087452b0495ee2 (127.27.161.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4b65187fc20346eda207dd58ee639ef2" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 36281 } } peers { permanent_uuid: "d102fadcc4b047ac9e652e1e0a4670a4" member_type: VOTER last_known_addr { host: "127.27.161.61" port: 43223 } attrs { promote: false } } peers { permanent_uuid: "1245ea4d6f734899a3087452b0495ee2" member_type: VOTER last_known_addr { host: "127.27.161.60" port: 43217 } attrs { promote: false } } }
I20260810 03:26:37.392951 6083 raft_consensus.cc:2932] T 00000000000000000000000000000000 P d102fadcc4b047ac9e652e1e0a4670a4 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 1245ea4d6f734899a3087452b0495ee2 (127.27.161.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4b65187fc20346eda207dd58ee639ef2" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 36281 } } peers { permanent_uuid: "d102fadcc4b047ac9e652e1e0a4670a4" member_type: VOTER last_known_addr { host: "127.27.161.61" port: 43223 } attrs { promote: false } } peers { permanent_uuid: "1245ea4d6f734899a3087452b0495ee2" member_type: VOTER last_known_addr { host: "127.27.161.60" port: 43217 } attrs { promote: false } } }
I20260810 03:26:37.410157 6431 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b65187fc20346eda207dd58ee639ef2 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "4b65187fc20346eda207dd58ee639ef2" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4b65187fc20346eda207dd58ee639ef2" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 36281 } } peers { permanent_uuid: "d102fadcc4b047ac9e652e1e0a4670a4" member_type: VOTER last_known_addr { host: "127.27.161.61" port: 43223 } attrs { promote: false } } peers { permanent_uuid: "1245ea4d6f734899a3087452b0495ee2" member_type: VOTER last_known_addr { host: "127.27.161.60" port: 43217 } attrs { promote: false } } }
I20260810 03:26:37.410293 6429 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b65187fc20346eda207dd58ee639ef2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4b65187fc20346eda207dd58ee639ef2" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4b65187fc20346eda207dd58ee639ef2" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 36281 } } peers { permanent_uuid: "d102fadcc4b047ac9e652e1e0a4670a4" member_type: VOTER last_known_addr { host: "127.27.161.61" port: 43223 } attrs { promote: false } } peers { permanent_uuid: "1245ea4d6f734899a3087452b0495ee2" member_type: VOTER last_known_addr { host: "127.27.161.60" port: 43217 } attrs { promote: false } } }
I20260810 03:26:37.411623 6431 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b65187fc20346eda207dd58ee639ef2 [sys.catalog]: This master's current role is: LEADER
I20260810 03:26:37.411830 6429 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b65187fc20346eda207dd58ee639ef2 [sys.catalog]: This master's current role is: LEADER
I20260810 03:26:37.415886 6359 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 1245ea4d6f734899a3087452b0495ee2 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 1245ea4d6f734899a3087452b0495ee2 (127.27.161.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4b65187fc20346eda207dd58ee639ef2" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 36281 } } peers { permanent_uuid: "d102fadcc4b047ac9e652e1e0a4670a4" member_type: VOTER last_known_addr { host: "127.27.161.61" port: 43223 } attrs { promote: false } } peers { permanent_uuid: "1245ea4d6f734899a3087452b0495ee2" member_type: VOTER last_known_addr { host: "127.27.161.60" port: 43217 } attrs { promote: false } } }
I20260810 03:26:37.423812 6421 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b65187fc20346eda207dd58ee639ef2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4b65187fc20346eda207dd58ee639ef2" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4b65187fc20346eda207dd58ee639ef2" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 36281 } } peers { permanent_uuid: "d102fadcc4b047ac9e652e1e0a4670a4" member_type: VOTER last_known_addr { host: "127.27.161.61" port: 43223 } attrs { promote: false } } peers { permanent_uuid: "1245ea4d6f734899a3087452b0495ee2" member_type: VOTER last_known_addr { host: "127.27.161.60" port: 43217 } attrs { promote: false } } }
I20260810 03:26:37.424937 6421 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b65187fc20346eda207dd58ee639ef2 [sys.catalog]: This master's current role is: LEADER
I20260810 03:26:37.431768 6438 sys_catalog.cc:455] T 00000000000000000000000000000000 P d102fadcc4b047ac9e652e1e0a4670a4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4b65187fc20346eda207dd58ee639ef2" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4b65187fc20346eda207dd58ee639ef2" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 36281 } } peers { permanent_uuid: "d102fadcc4b047ac9e652e1e0a4670a4" member_type: VOTER last_known_addr { host: "127.27.161.61" port: 43223 } attrs { promote: false } } peers { permanent_uuid: "1245ea4d6f734899a3087452b0495ee2" member_type: VOTER last_known_addr { host: "127.27.161.60" port: 43217 } attrs { promote: false } } }
I20260810 03:26:37.433408 6438 sys_catalog.cc:458] T 00000000000000000000000000000000 P d102fadcc4b047ac9e652e1e0a4670a4 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 03:26:37.437673 6405 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1245ea4d6f734899a3087452b0495ee2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4b65187fc20346eda207dd58ee639ef2" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4b65187fc20346eda207dd58ee639ef2" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 36281 } } peers { permanent_uuid: "d102fadcc4b047ac9e652e1e0a4670a4" member_type: VOTER last_known_addr { host: "127.27.161.61" port: 43223 } attrs { promote: false } } peers { permanent_uuid: "1245ea4d6f734899a3087452b0495ee2" member_type: VOTER last_known_addr { host: "127.27.161.60" port: 43217 } attrs { promote: false } } }
I20260810 03:26:37.438643 6405 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1245ea4d6f734899a3087452b0495ee2 [sys.catalog]: This master's current role is: FOLLOWER
W20260810 03:26:37.509059 6332 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:26:37.510198 6332 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:26:37.510576 6332 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:26:37.542706 6332 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:26:37.543113 6332 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:26:37.543673 6332 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.161.1
I20260810 03:26:37.588017 6332 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.161.20:43095
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786332336319544-28292-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786332336319544-28292-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786332336319544-28292-0/krb5kdc/oryx_127.27.161.1.keytab
--principal=oryx/127.27.161.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.161.1:36623
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786332336319544-28292-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.161.1
--webserver_port=39437
--tserver_master_addrs=127.27.161.62:36281,127.27.161.61:43223,127.27.161.60:43217
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.161.1
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786332336319544-28292-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:26:37.593832 6332 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:26:37.600178 6332 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:26:37.658316 6448 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:26:37.658339 6446 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:26:37.671281 6453 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:26:37.672533 6332 server_base.cc:1034] running on GCE node
I20260810 03:26:37.676715 6332 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:26:37.682973 6332 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:26:37.687023 6332 hybrid_clock.cc:648] HybridClock initialized: now 1786332397686588 us; error 305 us; skew 500 ppm
I20260810 03:26:37.689602 6332 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:26:37 dist-test-slave-vpvm krb5kdc[5096](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332397, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.161.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:26:37.711782 6332 init.cc:377] Logged in from keytab as oryx/127.27.161.1@KRBTEST.COM (short username oryx)
I20260810 03:26:37.719799 6332 webserver.cc:460] Webserver started at http://127.27.161.1:39437/ using document root <none> and password file <none>
I20260810 03:26:37.723255 6332 fs_manager.cc:359] Metadata directory not provided
I20260810 03:26:37.723454 6332 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:26:37.752604 6332 fs_manager.cc:709] Time spent opening directory manager: real 0.019s user 0.016s sys 0.001s
I20260810 03:26:37.768118 6460 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:26:37.775444 6332 fs_manager.cc:725] Time spent opening block manager: real 0.014s user 0.011s sys 0.000s
I20260810 03:26:37.776360 6332 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786332336319544-28292-0/minicluster-data/ts-0/data,/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786332336319544-28292-0/minicluster-data/ts-0/wal
uuid: "b7227f1049a6433489fae09b53501439"
format_stamp: "Formatted at 2026-08-10 03:26:33 on dist-test-slave-vpvm"
server_key: "c35f815820600d7c401954e211e9b3fb"
server_key_iv: "13fd7bbc96ac0a033eb66632ebd97c9f"
server_key_version: "encryptionkey@0"
I20260810 03:26:37.780901 6332 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786332336319544-28292-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786332336319544-28292-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786332336319544-28292-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:26:37.841552 6332 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:26:37.854943 6332 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:26:37.855723 6332 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:26:37.858495 6332 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:26:37.864323 6332 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:26:37.864925 6332 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.001s user 0.000s sys 0.000s
I20260810 03:26:37.865520 6332 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:26:37.865885 6332 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:26:38.001909 6332 rpc_server.cc:307] RPC server started. Bound to: 127.27.161.1:36623
I20260810 03:26:38.002378 6629 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.161.1:36623 every 8 connection(s)
I20260810 03:26:38.009235 6332 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786332336319544-28292-0/minicluster-data/ts-0/data/info.pb
I20260810 03:26:38.015945 28292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iBH5/build/asan/bin/kudu as pid 6332
I20260810 03:26:38.017119 28292 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iBH5/build/asan/bin/kudu pid 5454
Aug 10 03:26:38 dist-test-slave-vpvm krb5kdc[5096](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332392, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.27.161.2@KRBTEST.COM
Aug 10 03:26:38 dist-test-slave-vpvm krb5kdc[5096](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332397, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.161.1@KRBTEST.COM for oryx/127.27.161.62@KRBTEST.COM
Aug 10 03:26:38 dist-test-slave-vpvm krb5kdc[5096](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332397, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.161.1@KRBTEST.COM for oryx/127.27.161.61@KRBTEST.COM
Aug 10 03:26:38 dist-test-slave-vpvm krb5kdc[5096](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332397, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.161.1@KRBTEST.COM for oryx/127.27.161.60@KRBTEST.COM
I20260810 03:26:38.179731 6637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:38.023797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39458 (local address 127.27.161.2:43345)
0810 03:26:38.039091 (+ 15294us) server_negotiation.cc:206] Beginning negotiation
0810 03:26:38.039102 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 03:26:38.039142 (+ 40us) server_negotiation.cc:407] Connection header received
0810 03:26:38.039229 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:26:38.039239 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:26:38.040835 (+ 1596us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:26:38.041897 (+ 1062us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:26:38.058708 (+ 16811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:38.064505 (+ 5797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:38.074372 (+ 9867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:38.084763 (+ 10391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:38.085192 (+ 429us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:38.132866 (+ 47674us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:26:38.132967 (+ 101us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:26:38.133061 (+ 94us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:26:38.133309 (+ 248us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:26:38.158500 (+ 25191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:26:38.164294 (+ 5794us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:26:38.164355 (+ 61us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:26:38.164398 (+ 43us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:26:38.165288 (+ 890us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:26:38.168565 (+ 3277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:26:38.168615 (+ 50us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:26:38.168670 (+ 55us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:26:38.171211 (+ 2541us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:26:38.176266 (+ 5055us) server_negotiation.cc:1035] Waiting for connection context
0810 03:26:38.178085 (+ 1819us) server_negotiation.cc:299] Negotiation successful
0810 03:26:38.179276 (+ 1191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12319,"thread_start_us":14183,"threads_started":1}
I20260810 03:26:38.180763 6641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:38.049763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.161.1:55069 (local address 127.27.161.60:43217)
0810 03:26:38.052884 (+ 3121us) server_negotiation.cc:206] Beginning negotiation
0810 03:26:38.052932 (+ 48us) server_negotiation.cc:399] Waiting for connection header
0810 03:26:38.055698 (+ 2766us) server_negotiation.cc:407] Connection header received
0810 03:26:38.061322 (+ 5624us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:26:38.061366 (+ 44us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:26:38.062070 (+ 704us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:26:38.062218 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:26:38.077266 (+ 15048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:38.082623 (+ 5357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:38.103467 (+ 20844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:38.107766 (+ 4299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:38.108272 (+ 506us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:38.152242 (+ 43970us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:26:38.152301 (+ 59us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:26:38.152354 (+ 53us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:26:38.152702 (+ 348us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:26:38.172331 (+ 19629us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:26:38.175435 (+ 3104us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:26:38.175499 (+ 64us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:26:38.175527 (+ 28us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:26:38.176372 (+ 845us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:26:38.178216 (+ 1844us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:26:38.178222 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:26:38.178225 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:26:38.179259 (+ 1034us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:26:38.179400 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0810 03:26:38.180126 (+ 726us) server_negotiation.cc:299] Negotiation successful
0810 03:26:38.180396 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2482,"thread_start_us":1797,"threads_started":1}
I20260810 03:26:38.181419 5647 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:26:38.181860 6644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:38.050773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.60:43217 (local address 127.27.161.1:55069)
0810 03:26:38.054477 (+ 3704us) negotiation.cc:107] Waiting for socket to connect
0810 03:26:38.054611 (+ 134us) client_negotiation.cc:174] Beginning negotiation
0810 03:26:38.059303 (+ 4692us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:26:38.063360 (+ 4057us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:26:38.063464 (+ 104us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:26:38.069674 (+ 6210us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:26:38.076236 (+ 6562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:26:38.076351 (+ 115us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:26:38.083540 (+ 7189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:26:38.083585 (+ 45us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:26:38.102414 (+ 18829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:26:38.102435 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:26:38.109574 (+ 7139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:26:38.109582 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:26:38.114736 (+ 5154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:38.118670 (+ 3934us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:26:38.118845 (+ 175us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:26:38.150861 (+ 32016us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:26:38.172919 (+ 22058us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:26:38.172940 (+ 21us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:26:38.172958 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:26:38.174806 (+ 1848us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:26:38.177054 (+ 2248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:26:38.177072 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:26:38.177095 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:26:38.178050 (+ 955us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:26:38.179423 (+ 1373us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:26:38.179427 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:26:38.179610 (+ 183us) client_negotiation.cc:769] Sending connection context
0810 03:26:38.179747 (+ 137us) client_negotiation.cc:240] Negotiation successful
0810 03:26:38.180337 (+ 590us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2963,"mutex_wait_us":8313,"spinlock_wait_cycles":44928,"thread_start_us":2245,"threads_started":1}
I20260810 03:26:38.182065 6635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:38.021688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.161.1:56903 (local address 127.27.161.61:43223)
0810 03:26:38.045475 (+ 23787us) server_negotiation.cc:206] Beginning negotiation
0810 03:26:38.045534 (+ 59us) server_negotiation.cc:399] Waiting for connection header
0810 03:26:38.045647 (+ 113us) server_negotiation.cc:407] Connection header received
0810 03:26:38.061277 (+ 15630us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:26:38.061286 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:26:38.062070 (+ 784us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:26:38.062291 (+ 221us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:26:38.077944 (+ 15653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:38.084088 (+ 6144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:38.108364 (+ 24276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:38.112890 (+ 4526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:38.113827 (+ 937us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:38.145088 (+ 31261us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:26:38.145175 (+ 87us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:26:38.145222 (+ 47us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:26:38.145549 (+ 327us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:26:38.164137 (+ 18588us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:26:38.171305 (+ 7168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:26:38.171341 (+ 36us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:26:38.171365 (+ 24us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:26:38.172169 (+ 804us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:26:38.179499 (+ 7330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:26:38.179506 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:26:38.179509 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:26:38.180281 (+ 772us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:26:38.180428 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0810 03:26:38.181277 (+ 849us) server_negotiation.cc:299] Negotiation successful
0810 03:26:38.181574 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22822,"thread_start_us":22270,"threads_started":1}
I20260810 03:26:38.182538 6638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:38.024401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.61:43223 (local address 127.27.161.1:56903)
0810 03:26:38.038679 (+ 14278us) negotiation.cc:107] Waiting for socket to connect
0810 03:26:38.039028 (+ 349us) client_negotiation.cc:174] Beginning negotiation
0810 03:26:38.059303 (+ 20275us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:26:38.063533 (+ 4230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:26:38.063561 (+ 28us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:26:38.069674 (+ 6113us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:26:38.077016 (+ 7342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:26:38.077133 (+ 117us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:26:38.085192 (+ 8059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:26:38.085199 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:26:38.107718 (+ 22519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:26:38.107804 (+ 86us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:26:38.113896 (+ 6092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:26:38.113906 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:26:38.116308 (+ 2402us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:38.119711 (+ 3403us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:26:38.119944 (+ 233us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:26:38.142909 (+ 22965us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:26:38.165114 (+ 22205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:26:38.165153 (+ 39us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:26:38.165313 (+ 160us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:26:38.169584 (+ 4271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:26:38.177741 (+ 8157us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:26:38.177794 (+ 53us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:26:38.177838 (+ 44us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:26:38.178189 (+ 351us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:26:38.180522 (+ 2333us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:26:38.180528 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:26:38.180756 (+ 228us) client_negotiation.cc:769] Sending connection context
0810 03:26:38.181136 (+ 380us) client_negotiation.cc:240] Negotiation successful
0810 03:26:38.181460 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9412,"mutex_wait_us":3379,"thread_start_us":8930,"threads_started":1}
I20260810 03:26:38.183120 6640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:38.037575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:36281 (local address 127.27.161.1:35121)
0810 03:26:38.048853 (+ 11278us) negotiation.cc:107] Waiting for socket to connect
0810 03:26:38.048989 (+ 136us) client_negotiation.cc:174] Beginning negotiation
0810 03:26:38.059314 (+ 10325us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:26:38.063352 (+ 4038us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:26:38.063444 (+ 92us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:26:38.069636 (+ 6192us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:26:38.076236 (+ 6600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:26:38.076351 (+ 115us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:26:38.091169 (+ 14818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:26:38.091178 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:26:38.100101 (+ 8923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:26:38.100242 (+ 141us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:26:38.110686 (+ 10444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:26:38.110724 (+ 38us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:26:38.112806 (+ 2082us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:38.121083 (+ 8277us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:26:38.121115 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:26:38.137164 (+ 16049us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:26:38.166910 (+ 29746us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:26:38.166917 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:26:38.166920 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:26:38.170502 (+ 3582us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:26:38.172683 (+ 2181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:26:38.172688 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:26:38.172691 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:26:38.174339 (+ 1648us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:26:38.177347 (+ 3008us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:26:38.177359 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:26:38.178948 (+ 1589us) client_negotiation.cc:769] Sending connection context
0810 03:26:38.179571 (+ 623us) client_negotiation.cc:240] Negotiation successful
0810 03:26:38.182686 (+ 3115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10475,"mutex_wait_us":6686,"thread_start_us":3936,"threads_started":1}
I20260810 03:26:38.184430 6630 heartbeater.cc:347] Connected to a master server at 127.27.161.60:43217
I20260810 03:26:38.184828 6639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:38.037717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.161.1:35121 (local address 127.27.161.62:36281)
0810 03:26:38.057178 (+ 19461us) server_negotiation.cc:206] Beginning negotiation
0810 03:26:38.057183 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:26:38.057233 (+ 50us) server_negotiation.cc:407] Connection header received
0810 03:26:38.061868 (+ 4635us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:26:38.061905 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:26:38.062111 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:26:38.064237 (+ 2126us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:26:38.078446 (+ 14209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:38.090559 (+ 12113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:38.100867 (+ 10308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:38.104532 (+ 3665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:38.105162 (+ 630us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:38.138723 (+ 33561us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:26:38.138790 (+ 67us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:26:38.138856 (+ 66us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:26:38.139331 (+ 475us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:26:38.166264 (+ 26933us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:26:38.171285 (+ 5021us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:26:38.171294 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:26:38.171298 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:26:38.172113 (+ 815us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:26:38.174791 (+ 2678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:26:38.174819 (+ 28us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:26:38.174844 (+ 25us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:26:38.176804 (+ 1960us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:26:38.177457 (+ 653us) server_negotiation.cc:1035] Waiting for connection context
0810 03:26:38.183878 (+ 6421us) server_negotiation.cc:299] Negotiation successful
0810 03:26:38.184276 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19084,"thread_start_us":18631,"threads_started":1}
I20260810 03:26:38.184890 6630 heartbeater.cc:464] Registering TS with master...
I20260810 03:26:38.186193 6631 heartbeater.cc:347] Connected to a master server at 127.27.161.62:36281
I20260810 03:26:38.186353 6632 heartbeater.cc:347] Connected to a master server at 127.27.161.61:43223
I20260810 03:26:38.186460 6632 heartbeater.cc:464] Registering TS with master...
I20260810 03:26:38.186643 6631 heartbeater.cc:464] Registering TS with master...
I20260810 03:26:38.187443 6630 heartbeater.cc:511] Master 127.27.161.60:43217 requested a full tablet report, sending...
I20260810 03:26:38.187677 6631 heartbeater.cc:511] Master 127.27.161.62:36281 requested a full tablet report, sending...
I20260810 03:26:38.188458 6632 heartbeater.cc:511] Master 127.27.161.61:43223 requested a full tablet report, sending...
I20260810 03:26:38.189987 5158 ts_manager.cc:194] Re-registered known tserver with Master: b7227f1049a6433489fae09b53501439 (127.27.161.1:36623)
I20260810 03:26:38.190474 6346 ts_manager.cc:194] Registered new tserver with Master: b7227f1049a6433489fae09b53501439 (127.27.161.1:36623)
I20260810 03:26:38.190470 6069 ts_manager.cc:194] Registered new tserver with Master: b7227f1049a6433489fae09b53501439 (127.27.161.1:36623)
I20260810 03:26:38.191205 5158 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.27.161.1@KRBTEST.COM'} at 127.27.161.1:35121
I20260810 03:26:38.273566 28292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iBH5/build/asan/bin/kudu with pid 5454
I20260810 03:26:38.298442 28292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iBH5/build/asan/bin/kudu
/tmp/dist-test-task24iBH5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786332336319544-28292-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786332336319544-28292-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786332336319544-28292-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786332336319544-28292-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.161.2:43345
--local_ip_for_outbound_sockets=127.27.161.2
--tserver_master_addrs=127.27.161.62:36281
--webserver_port=33517
--webserver_interface=127.27.161.2
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786332336319544-28292-0/krb5kdc/oryx_127.27.161.2.keytab
--principal=oryx/127.27.161.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.161.20:43095
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.27.161.62:36281,127.27.161.61:43223
--tserver_master_addrs=127.27.161.62:36281,127.27.161.61:43223,127.27.161.60:43217 with env {KRB5CCNAME=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786332336319544-28292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786332336319544-28292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786332336319544-28292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:26:38.653301 6646 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:26:38.653621 6646 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:26:38.653762 6646 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:26:38.661266 6646 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:26:38.661408 6646 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:26:38.661563 6646 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.161.2
I20260810 03:26:38.671783 6646 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.161.20:43095
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786332336319544-28292-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786332336319544-28292-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786332336319544-28292-0/krb5kdc/oryx_127.27.161.2.keytab
--principal=oryx/127.27.161.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.161.2:43345
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786332336319544-28292-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.161.2
--webserver_port=33517
--tserver_master_addrs=127.27.161.62:36281,127.27.161.61:43223,127.27.161.60:43217
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.161.2
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786332336319544-28292-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:26:38.673169 6646 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:26:38.674530 6646 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:26:38.688218 6652 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:26:38.688253 6660 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:26:38.688293 6654 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:26:38.689237 6646 server_base.cc:1034] running on GCE node
I20260810 03:26:38.689888 6646 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:26:38.690917 6646 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:26:38.692142 6646 hybrid_clock.cc:648] HybridClock initialized: now 1786332398692067 us; error 55 us; skew 500 ppm
I20260810 03:26:38.692498 6646 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:26:38 dist-test-slave-vpvm krb5kdc[5096](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332398, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.161.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:26:38.695757 6646 init.cc:377] Logged in from keytab as oryx/127.27.161.2@KRBTEST.COM (short username oryx)
I20260810 03:26:38.697209 6646 webserver.cc:460] Webserver started at http://127.27.161.2:33517/ using document root <none> and password file <none>
I20260810 03:26:38.697908 6646 fs_manager.cc:359] Metadata directory not provided
I20260810 03:26:38.698083 6646 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:26:38.703474 6646 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.004s
I20260810 03:26:38.706197 6670 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:26:38.707458 6646 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260810 03:26:38.707613 6646 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786332336319544-28292-0/minicluster-data/ts-1/data,/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786332336319544-28292-0/minicluster-data/ts-1/wal
uuid: "b851ca7ac5ca41af917e9e02fb356ce8"
format_stamp: "Formatted at 2026-08-10 03:26:34 on dist-test-slave-vpvm"
server_key: "d2a274c144128afaa4de0942893dd2b8"
server_key_iv: "bad13effa24215efa14c62d2f917a0fd"
server_key_version: "encryptionkey@0"
I20260810 03:26:38.708559 6646 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786332336319544-28292-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786332336319544-28292-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786332336319544-28292-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:26:38.732787 6646 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:26:38.736847 6646 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:26:38.737114 6646 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:26:38.738184 6646 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:26:38.739588 6646 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:26:38.739681 6646 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:26:38.739764 6646 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:26:38.739817 6646 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:26:38.772054 6646 rpc_server.cc:307] RPC server started. Bound to: 127.27.161.2:43345
I20260810 03:26:38.772105 6842 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.161.2:43345 every 8 connection(s)
I20260810 03:26:38.773531 6646 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786332336319544-28292-0/minicluster-data/ts-1/data/info.pb
I20260810 03:26:38.776981 28292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iBH5/build/asan/bin/kudu as pid 6646
I20260810 03:26:38.777280 28292 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iBH5/build/asan/bin/kudu pid 5663
Aug 10 03:26:38 dist-test-slave-vpvm krb5kdc[5096](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332398, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.161.2@KRBTEST.COM for oryx/127.27.161.62@KRBTEST.COM
Aug 10 03:26:38 dist-test-slave-vpvm krb5kdc[5096](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332398, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.161.2@KRBTEST.COM for oryx/127.27.161.61@KRBTEST.COM
Aug 10 03:26:38 dist-test-slave-vpvm krb5kdc[5096](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332398, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.161.2@KRBTEST.COM for oryx/127.27.161.60@KRBTEST.COM
Aug 10 03:26:38 dist-test-slave-vpvm krb5kdc[5096](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332392, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.27.161.3@KRBTEST.COM
I20260810 03:26:38.802994 6847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:38.776664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.161.2:45741 (local address 127.27.161.62:36281)
0810 03:26:38.777193 (+ 529us) server_negotiation.cc:206] Beginning negotiation
0810 03:26:38.777199 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:26:38.778337 (+ 1138us) server_negotiation.cc:407] Connection header received
0810 03:26:38.781418 (+ 3081us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:26:38.781425 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:26:38.781513 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:26:38.781582 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:26:38.783532 (+ 1950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:38.784669 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:38.786754 (+ 2085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:38.787208 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:38.787345 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:38.793863 (+ 6518us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:26:38.793871 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:26:38.793876 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:26:38.793927 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:26:38.798287 (+ 4360us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:26:38.799544 (+ 1257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:26:38.799551 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:26:38.799555 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:26:38.799742 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:26:38.800434 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:26:38.800440 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:26:38.800444 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:26:38.800898 (+ 454us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:26:38.801063 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0810 03:26:38.802336 (+ 1273us) server_negotiation.cc:299] Negotiation successful
0810 03:26:38.802605 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358,"thread_start_us":230,"threads_started":1}
I20260810 03:26:38.804026 6853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:38.779975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.161.2:57367 (local address 127.27.161.61:43223)
0810 03:26:38.784517 (+ 4542us) server_negotiation.cc:206] Beginning negotiation
0810 03:26:38.784523 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:26:38.784541 (+ 18us) server_negotiation.cc:407] Connection header received
0810 03:26:38.784600 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:26:38.784604 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:26:38.784690 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:26:38.785743 (+ 1053us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:26:38.787153 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:38.787959 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:38.790507 (+ 2548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:38.791197 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:38.792033 (+ 836us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:38.795613 (+ 3580us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:26:38.795620 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:26:38.795625 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:26:38.795682 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:26:38.799331 (+ 3649us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:26:38.800257 (+ 926us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:26:38.800263 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:26:38.800267 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:26:38.800544 (+ 277us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:26:38.801138 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:26:38.801143 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:26:38.801146 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:26:38.801529 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:26:38.803261 (+ 1732us) server_negotiation.cc:1035] Waiting for connection context
0810 03:26:38.803410 (+ 149us) server_negotiation.cc:299] Negotiation successful
0810 03:26:38.803653 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4417,"thread_start_us":4310,"threads_started":1}
I20260810 03:26:38.804561 6851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:38.779005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.61:43223 (local address 127.27.161.2:57367)
0810 03:26:38.779615 (+ 610us) negotiation.cc:107] Waiting for socket to connect
0810 03:26:38.779643 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0810 03:26:38.780586 (+ 943us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:26:38.785791 (+ 5205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:26:38.785796 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:26:38.786637 (+ 841us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:26:38.786959 (+ 322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:26:38.786973 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:26:38.788106 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:26:38.788111 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:26:38.790285 (+ 2174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:26:38.790303 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:26:38.791424 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:26:38.791432 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:26:38.791625 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:38.792352 (+ 727us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:26:38.792377 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:26:38.795350 (+ 2973us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:26:38.799504 (+ 4154us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:26:38.799510 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:26:38.799514 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:26:38.800062 (+ 548us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:26:38.800691 (+ 629us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:26:38.800696 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:26:38.800699 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:26:38.800919 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:26:38.801675 (+ 756us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:26:38.801680 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:26:38.801848 (+ 168us) client_negotiation.cc:769] Sending connection context
0810 03:26:38.802114 (+ 266us) client_negotiation.cc:240] Negotiation successful
0810 03:26:38.802793 (+ 679us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":466,"mutex_wait_us":34,"spinlock_wait_cycles":39040,"thread_start_us":1327,"threads_started":1}
I20260810 03:26:38.806094 6848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:38.777303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:36281 (local address 127.27.161.2:45741)
0810 03:26:38.778167 (+ 864us) negotiation.cc:107] Waiting for socket to connect
0810 03:26:38.778220 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0810 03:26:38.780582 (+ 2362us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:26:38.782012 (+ 1430us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:26:38.782022 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:26:38.782612 (+ 590us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:26:38.783339 (+ 727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:26:38.783360 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:26:38.784858 (+ 1498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:26:38.784862 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:26:38.786609 (+ 1747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:26:38.786622 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:26:38.787459 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:26:38.787467 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:26:38.787794 (+ 327us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:38.788684 (+ 890us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:26:38.788724 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:26:38.792468 (+ 3744us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:26:38.798592 (+ 6124us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:26:38.798606 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:26:38.798631 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:26:38.799339 (+ 708us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:26:38.799933 (+ 594us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:26:38.799939 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:26:38.799942 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:26:38.800272 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:26:38.801183 (+ 911us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:26:38.801193 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:26:38.801751 (+ 558us) client_negotiation.cc:769] Sending connection context
0810 03:26:38.802128 (+ 377us) client_negotiation.cc:240] Negotiation successful
0810 03:26:38.802956 (+ 828us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":513,"mutex_wait_us":180,"thread_start_us":303,"threads_started":1}
I20260810 03:26:38.806613 6845 heartbeater.cc:347] Connected to a master server at 127.27.161.61:43223
I20260810 03:26:38.806984 6845 heartbeater.cc:464] Registering TS with master...
I20260810 03:26:38.807178 6844 heartbeater.cc:347] Connected to a master server at 127.27.161.62:36281
I20260810 03:26:38.807322 6844 heartbeater.cc:464] Registering TS with master...
I20260810 03:26:38.808175 6844 heartbeater.cc:511] Master 127.27.161.62:36281 requested a full tablet report, sending...
I20260810 03:26:38.808271 6845 heartbeater.cc:511] Master 127.27.161.61:43223 requested a full tablet report, sending...
I20260810 03:26:38.809202 6856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:38.781198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.60:43217 (local address 127.27.161.2:54283)
0810 03:26:38.783942 (+ 2744us) negotiation.cc:107] Waiting for socket to connect
0810 03:26:38.783963 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0810 03:26:38.784083 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:26:38.785942 (+ 1859us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:26:38.785969 (+ 27us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:26:38.788916 (+ 2947us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:26:38.789406 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:26:38.789419 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:26:38.791143 (+ 1724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:26:38.791161 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:26:38.793038 (+ 1877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:26:38.793054 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:26:38.794707 (+ 1653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:26:38.794714 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:26:38.794927 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:38.795462 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:26:38.795480 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:26:38.798055 (+ 2575us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:26:38.804694 (+ 6639us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:26:38.804699 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:26:38.804703 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:26:38.805930 (+ 1227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:26:38.806641 (+ 711us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:26:38.806645 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:26:38.806647 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:26:38.806861 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:26:38.807642 (+ 781us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:26:38.807648 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:26:38.807935 (+ 287us) client_negotiation.cc:769] Sending connection context
0810 03:26:38.808139 (+ 204us) client_negotiation.cc:240] Negotiation successful
0810 03:26:38.808495 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2618,"mutex_wait_us":68,"thread_start_us":2463,"threads_started":1}
I20260810 03:26:38.809778 6854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:38.780994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.161.2:54283 (local address 127.27.161.60:43217)
0810 03:26:38.781425 (+ 431us) server_negotiation.cc:206] Beginning negotiation
0810 03:26:38.781430 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:26:38.784522 (+ 3092us) server_negotiation.cc:407] Connection header received
0810 03:26:38.784623 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:26:38.784629 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:26:38.784732 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:26:38.784862 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:26:38.789613 (+ 4751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:38.790937 (+ 1324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:38.793234 (+ 2297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:38.793894 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:38.794652 (+ 758us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:38.798334 (+ 3682us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:26:38.798342 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:26:38.798347 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:26:38.798394 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:26:38.804452 (+ 6058us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:26:38.806187 (+ 1735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:26:38.806196 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:26:38.806200 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:26:38.806458 (+ 258us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:26:38.807003 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:26:38.807009 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:26:38.807013 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:26:38.807478 (+ 465us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:26:38.807640 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0810 03:26:38.809121 (+ 1481us) server_negotiation.cc:299] Negotiation successful
0810 03:26:38.809403 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":314,"thread_start_us":215,"threads_started":1}
I20260810 03:26:38.810262 5158 ts_manager.cc:194] Re-registered known tserver with Master: b851ca7ac5ca41af917e9e02fb356ce8 (127.27.161.2:43345)
I20260810 03:26:38.810423 6069 ts_manager.cc:194] Registered new tserver with Master: b851ca7ac5ca41af917e9e02fb356ce8 (127.27.161.2:43345)
I20260810 03:26:38.810922 6843 heartbeater.cc:347] Connected to a master server at 127.27.161.60:43217
I20260810 03:26:38.811151 6843 heartbeater.cc:464] Registering TS with master...
I20260810 03:26:38.811455 5158 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.27.161.2@KRBTEST.COM'} at 127.27.161.2:45741
I20260810 03:26:38.811612 6843 heartbeater.cc:511] Master 127.27.161.60:43217 requested a full tablet report, sending...
I20260810 03:26:38.813040 6852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:38.779591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57840 (local address 127.27.161.3:45853)
0810 03:26:38.780207 (+ 616us) server_negotiation.cc:206] Beginning negotiation
0810 03:26:38.780217 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 03:26:38.785593 (+ 5376us) server_negotiation.cc:407] Connection header received
0810 03:26:38.785717 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:26:38.785726 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:26:38.786027 (+ 301us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:26:38.786208 (+ 181us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:26:38.788379 (+ 2171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:38.790026 (+ 1647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:38.791889 (+ 1863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:38.792815 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:38.792891 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:38.801134 (+ 8243us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:26:38.801158 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:26:38.801181 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:26:38.801229 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:26:38.806008 (+ 4779us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:26:38.809779 (+ 3771us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:26:38.809789 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:26:38.809793 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:26:38.810118 (+ 325us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:26:38.810726 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:26:38.810731 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:26:38.810735 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:26:38.811249 (+ 514us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:26:38.811599 (+ 350us) server_negotiation.cc:1035] Waiting for connection context
0810 03:26:38.812290 (+ 691us) server_negotiation.cc:299] Negotiation successful
0810 03:26:38.812628 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":430,"thread_start_us":543,"threads_started":1}
I20260810 03:26:38.813858 6346 ts_manager.cc:194] Registered new tserver with Master: b851ca7ac5ca41af917e9e02fb356ce8 (127.27.161.2:43345)
I20260810 03:26:38.814915 5836 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:26:38.900919 28292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iBH5/build/asan/bin/kudu with pid 5663
I20260810 03:26:38.924387 28292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iBH5/build/asan/bin/kudu
/tmp/dist-test-task24iBH5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786332336319544-28292-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786332336319544-28292-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786332336319544-28292-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786332336319544-28292-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.161.3:45853
--local_ip_for_outbound_sockets=127.27.161.3
--tserver_master_addrs=127.27.161.62:36281
--webserver_port=40551
--webserver_interface=127.27.161.3
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786332336319544-28292-0/krb5kdc/oryx_127.27.161.3.keytab
--principal=oryx/127.27.161.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.161.20:43095
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.27.161.62:36281,127.27.161.61:43223
--tserver_master_addrs=127.27.161.62:36281,127.27.161.61:43223,127.27.161.60:43217 with env {KRB5CCNAME=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786332336319544-28292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786332336319544-28292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786332336319544-28292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 03:26:39.193456 6631 heartbeater.cc:503] Master 127.27.161.62:36281 was elected leader, sending a full tablet report...
W20260810 03:26:39.292162 6861 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:26:39.292434 6861 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:26:39.292567 6861 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:26:39.300019 6861 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:26:39.300145 6861 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:26:39.300298 6861 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.161.3
I20260810 03:26:39.310742 6861 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.161.20:43095
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786332336319544-28292-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786332336319544-28292-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786332336319544-28292-0/krb5kdc/oryx_127.27.161.3.keytab
--principal=oryx/127.27.161.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.161.3:45853
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786332336319544-28292-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.161.3
--webserver_port=40551
--tserver_master_addrs=127.27.161.62:36281,127.27.161.61:43223,127.27.161.60:43217
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.161.3
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786332336319544-28292-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:26:39.312176 6861 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:26:39.313642 6861 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:26:39.328229 6868 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:26:39.328200 6872 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:26:39.328161 6869 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:26:39.329986 6861 server_base.cc:1034] running on GCE node
I20260810 03:26:39.330714 6861 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:26:39.331808 6861 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:26:39.333076 6861 hybrid_clock.cc:648] HybridClock initialized: now 1786332399333008 us; error 49 us; skew 500 ppm
I20260810 03:26:39.333472 6861 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:26:39 dist-test-slave-vpvm krb5kdc[5096](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332399, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.161.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:26:39.336954 6861 init.cc:377] Logged in from keytab as oryx/127.27.161.3@KRBTEST.COM (short username oryx)
I20260810 03:26:39.338814 6861 webserver.cc:460] Webserver started at http://127.27.161.3:40551/ using document root <none> and password file <none>
I20260810 03:26:39.339639 6861 fs_manager.cc:359] Metadata directory not provided
I20260810 03:26:39.339802 6861 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:26:39.346115 6861 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260810 03:26:39.349054 6882 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:26:39.350657 6861 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 03:26:39.350842 6861 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786332336319544-28292-0/minicluster-data/ts-2/data,/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786332336319544-28292-0/minicluster-data/ts-2/wal
uuid: "05f07b3f00144d75b6c6fc9a40632f17"
format_stamp: "Formatted at 2026-08-10 03:26:34 on dist-test-slave-vpvm"
server_key: "c71623682b543b69a57bb03ef3329687"
server_key_iv: "a8d2d31b59e20e9d16f868ed791d7ac0"
server_key_version: "encryptionkey@0"
I20260810 03:26:39.351903 6861 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786332336319544-28292-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786332336319544-28292-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786332336319544-28292-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:26:39.376655 6861 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:26:39.381335 6861 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:26:39.381649 6861 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:26:39.382790 6861 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:26:39.384460 6861 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:26:39.384567 6861 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:26:39.384680 6861 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:26:39.384759 6861 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:26:39.423636 6861 rpc_server.cc:307] RPC server started. Bound to: 127.27.161.3:45853
I20260810 03:26:39.423712 7052 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.161.3:45853 every 8 connection(s)
I20260810 03:26:39.426185 6861 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786332336319544-28292-0/minicluster-data/ts-2/data/info.pb
I20260810 03:26:39.435294 28292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iBH5/build/asan/bin/kudu as pid 6861
Aug 10 03:26:39 dist-test-slave-vpvm krb5kdc[5096](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332399, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.161.3@KRBTEST.COM for oryx/127.27.161.60@KRBTEST.COM
Aug 10 03:26:39 dist-test-slave-vpvm krb5kdc[5096](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332399, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.161.3@KRBTEST.COM for oryx/127.27.161.62@KRBTEST.COM
I20260810 03:26:39.516395 7058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:39.429684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.161.3:46575 (local address 127.27.161.60:43217)
0810 03:26:39.458808 (+ 29124us) server_negotiation.cc:206] Beginning negotiation
0810 03:26:39.458816 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 03:26:39.458843 (+ 27us) server_negotiation.cc:407] Connection header received
0810 03:26:39.458929 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:26:39.458935 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:26:39.459032 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:26:39.459184 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:26:39.474613 (+ 15429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:39.475807 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:39.482261 (+ 6454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:39.483398 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:39.484099 (+ 701us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:39.507319 (+ 23220us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:26:39.507327 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:26:39.507332 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:26:39.507383 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:26:39.511360 (+ 3977us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:26:39.512774 (+ 1414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:26:39.512781 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:26:39.512784 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:26:39.513017 (+ 233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:26:39.513676 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:26:39.513681 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:26:39.513684 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:26:39.514086 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:26:39.514231 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0810 03:26:39.515577 (+ 1346us) server_negotiation.cc:299] Negotiation successful
0810 03:26:39.515866 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28948,"thread_start_us":29628,"threads_started":1}
I20260810 03:26:39.519284 7059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:39.430495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.60:43217 (local address 127.27.161.3:46575)
0810 03:26:39.440697 (+ 10202us) negotiation.cc:107] Waiting for socket to connect
0810 03:26:39.440776 (+ 79us) client_negotiation.cc:174] Beginning negotiation
0810 03:26:39.443600 (+ 2824us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:26:39.464829 (+ 21229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:26:39.464847 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:26:39.465833 (+ 986us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:26:39.466880 (+ 1047us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:26:39.466914 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:26:39.476123 (+ 9209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:26:39.476128 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:26:39.478786 (+ 2658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:26:39.478805 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:26:39.484716 (+ 5911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:26:39.484725 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:26:39.485143 (+ 418us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:39.500189 (+ 15046us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:26:39.500227 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:26:39.506969 (+ 6742us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:26:39.511551 (+ 4582us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:26:39.511562 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:26:39.511583 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:26:39.512609 (+ 1026us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:26:39.513247 (+ 638us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:26:39.513253 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:26:39.513257 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:26:39.513521 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:26:39.514343 (+ 822us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:26:39.514352 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:26:39.515100 (+ 748us) client_negotiation.cc:769] Sending connection context
0810 03:26:39.516986 (+ 1886us) client_negotiation.cc:240] Negotiation successful
0810 03:26:39.517643 (+ 657us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9774,"thread_start_us":13487,"threads_started":1}
I20260810 03:26:39.522897 7053 heartbeater.cc:347] Connected to a master server at 127.27.161.60:43217
I20260810 03:26:39.523351 7053 heartbeater.cc:464] Registering TS with master...
I20260810 03:26:39.524492 7053 heartbeater.cc:511] Master 127.27.161.60:43217 requested a full tablet report, sending...
I20260810 03:26:39.526804 6346 ts_manager.cc:194] Registered new tserver with Master: 05f07b3f00144d75b6c6fc9a40632f17 (127.27.161.3:45853)
I20260810 03:26:39.533143 7065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:39.451458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:36281 (local address 127.27.161.3:40799)
0810 03:26:39.459220 (+ 7762us) negotiation.cc:107] Waiting for socket to connect
0810 03:26:39.459242 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0810 03:26:39.459364 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:26:39.473566 (+ 14202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:26:39.473573 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:26:39.474381 (+ 808us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:26:39.474778 (+ 397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:26:39.474795 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:26:39.488929 (+ 14134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:26:39.488937 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:26:39.501442 (+ 12505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:26:39.501465 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:26:39.503922 (+ 2457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:26:39.503930 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:26:39.504114 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:39.508529 (+ 4415us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:26:39.508556 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:26:39.514489 (+ 5933us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:26:39.529873 (+ 15384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:26:39.529876 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:26:39.529878 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:26:39.530615 (+ 737us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:26:39.531116 (+ 501us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:26:39.531136 (+ 20us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:26:39.531140 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:26:39.531302 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:26:39.532145 (+ 843us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:26:39.532150 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:26:39.532321 (+ 171us) client_negotiation.cc:769] Sending connection context
0810 03:26:39.532470 (+ 149us) client_negotiation.cc:240] Negotiation successful
0810 03:26:39.532795 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7611,"thread_start_us":7941,"threads_started":1}
Aug 10 03:26:39 dist-test-slave-vpvm krb5kdc[5096](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332399, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.161.3@KRBTEST.COM for oryx/127.27.161.61@KRBTEST.COM
I20260810 03:26:39.534180 7063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:39.445424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.161.3:40799 (local address 127.27.161.62:36281)
0810 03:26:39.472958 (+ 27534us) server_negotiation.cc:206] Beginning negotiation
0810 03:26:39.472966 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 03:26:39.472990 (+ 24us) server_negotiation.cc:407] Connection header received
0810 03:26:39.473068 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:26:39.473074 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:26:39.473198 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:26:39.473341 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:26:39.477924 (+ 4583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:39.479057 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:39.501675 (+ 22618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:39.502566 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:39.502706 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:39.514983 (+ 12277us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:26:39.514991 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:26:39.514996 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:26:39.515044 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:26:39.529719 (+ 14675us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:26:39.530771 (+ 1052us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:26:39.530776 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:26:39.530780 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:26:39.530955 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:26:39.531439 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:26:39.531445 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:26:39.531448 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:26:39.531912 (+ 464us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:26:39.532056 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0810 03:26:39.533499 (+ 1443us) server_negotiation.cc:299] Negotiation successful
0810 03:26:39.533733 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27387,"thread_start_us":29482,"threads_started":1}
I20260810 03:26:39.535173 7054 heartbeater.cc:347] Connected to a master server at 127.27.161.62:36281
I20260810 03:26:39.535313 7054 heartbeater.cc:464] Registering TS with master...
I20260810 03:26:39.535924 7054 heartbeater.cc:511] Master 127.27.161.62:36281 requested a full tablet report, sending...
I20260810 03:26:39.537184 5158 ts_manager.cc:194] Re-registered known tserver with Master: 05f07b3f00144d75b6c6fc9a40632f17 (127.27.161.3:45853)
I20260810 03:26:39.538394 5158 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.27.161.3@KRBTEST.COM'} at 127.27.161.3:40799
I20260810 03:26:39.541796 7077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:39.487962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32960 (local address 127.27.161.60:43217)
0810 03:26:39.505731 (+ 17769us) server_negotiation.cc:206] Beginning negotiation
0810 03:26:39.505738 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:26:39.515241 (+ 9503us) server_negotiation.cc:407] Connection header received
0810 03:26:39.515305 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:26:39.515310 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:26:39.515395 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:26:39.515489 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:26:39.520288 (+ 4799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:39.521783 (+ 1495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:39.526560 (+ 4777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:39.527338 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:39.527447 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:39.534008 (+ 6561us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:26:39.534017 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:26:39.534022 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:26:39.534074 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:26:39.538021 (+ 3947us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:26:39.538816 (+ 795us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:26:39.538823 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:26:39.538826 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:26:39.538989 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:26:39.539428 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:26:39.539434 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:26:39.539437 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:26:39.539784 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:26:39.540025 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0810 03:26:39.541138 (+ 1113us) server_negotiation.cc:299] Negotiation successful
0810 03:26:39.541386 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17596,"thread_start_us":17765,"threads_started":1}
I20260810 03:26:39.543676 7074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:39.479498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50852 (local address 127.27.161.61:43223)
0810 03:26:39.498948 (+ 19450us) server_negotiation.cc:206] Beginning negotiation
0810 03:26:39.498954 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:26:39.498973 (+ 19us) server_negotiation.cc:407] Connection header received
0810 03:26:39.499021 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:26:39.499027 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:26:39.499124 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:26:39.499246 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:26:39.512028 (+ 12782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:39.513591 (+ 1563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:39.518196 (+ 4605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:39.518849 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:39.518984 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:39.524724 (+ 5740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:26:39.524732 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:26:39.524736 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:26:39.524778 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:26:39.528392 (+ 3614us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:26:39.529498 (+ 1106us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:26:39.529503 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:26:39.529507 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:26:39.529706 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:26:39.534444 (+ 4738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:26:39.534449 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:26:39.534452 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:26:39.542074 (+ 7622us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:26:39.542288 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0810 03:26:39.543151 (+ 863us) server_negotiation.cc:299] Negotiation successful
0810 03:26:39.543411 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19295,"thread_start_us":18496,"threads_started":1}
I20260810 03:26:39.546379 7073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:39.475144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37096 (local address 127.27.161.62:36281)
0810 03:26:39.492469 (+ 17325us) server_negotiation.cc:206] Beginning negotiation
0810 03:26:39.492477 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 03:26:39.492502 (+ 25us) server_negotiation.cc:407] Connection header received
0810 03:26:39.492577 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:26:39.492583 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:26:39.492725 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:26:39.492863 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:26:39.506784 (+ 13921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:39.508167 (+ 1383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:39.510147 (+ 1980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:39.510869 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:39.511001 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:39.519344 (+ 8343us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:26:39.519352 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:26:39.519357 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:26:39.519406 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:26:39.532113 (+ 12707us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:26:39.543914 (+ 11801us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:26:39.543919 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:26:39.543922 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:26:39.544077 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:26:39.544894 (+ 817us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:26:39.544898 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:26:39.544900 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:26:39.545203 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:26:39.545383 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0810 03:26:39.545796 (+ 413us) server_negotiation.cc:299] Negotiation successful
0810 03:26:39.546068 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17114,"thread_start_us":19683,"threads_started":1}
I20260810 03:26:39.547693 7072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:39.472339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.61:43223 (local address 127.27.161.3:58943)
0810 03:26:39.495629 (+ 23290us) negotiation.cc:107] Waiting for socket to connect
0810 03:26:39.495653 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0810 03:26:39.495900 (+ 247us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:26:39.499569 (+ 3669us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:26:39.499574 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:26:39.500228 (+ 654us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:26:39.500704 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:26:39.500727 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:26:39.504644 (+ 3917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:26:39.504649 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:26:39.506381 (+ 1732us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:26:39.506399 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:26:39.511297 (+ 4898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:26:39.511305 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:26:39.511488 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:39.521410 (+ 9922us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:26:39.521442 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:26:39.534169 (+ 12727us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:26:39.542413 (+ 8244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:26:39.542417 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:26:39.542420 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:26:39.542940 (+ 520us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:26:39.544577 (+ 1637us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:26:39.544582 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:26:39.544586 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:26:39.544774 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:26:39.546582 (+ 1808us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:26:39.546587 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:26:39.546758 (+ 171us) client_negotiation.cc:769] Sending connection context
0810 03:26:39.546918 (+ 160us) client_negotiation.cc:240] Negotiation successful
0810 03:26:39.547255 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23119,"thread_start_us":23617,"threads_started":1}
I20260810 03:26:39.548435 7070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:39.460362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.161.3:58943 (local address 127.27.161.61:43223)
0810 03:26:39.472018 (+ 11656us) server_negotiation.cc:206] Beginning negotiation
0810 03:26:39.472027 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 03:26:39.495768 (+ 23741us) server_negotiation.cc:407] Connection header received
0810 03:26:39.498248 (+ 2480us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:26:39.498257 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:26:39.498381 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:26:39.498472 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:26:39.500970 (+ 2498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:39.502187 (+ 1217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:39.506759 (+ 4572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:39.507550 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:39.507681 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:39.534983 (+ 27302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:26:39.534991 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:26:39.534996 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:26:39.538776 (+ 3780us) spinlock_profiling.cc:243] Waited 3.14 ms on lock 0x60c0000347b8. stack: 00007fecc37cd66e 00007fecc37cd430 00007fecc28545ed 00000000008ea064 00000000008ea02d 00000000008e99fa 00007fecc37c30fd 00007fecc37c2d1e 00007fecca2da664 00007fecca2dcb75 00007fecca2f670e 00007fecca2f001e 00007fecca2e98a5 00007fecca257ea5 00007fecca256eff 00007fecca28a82f
0810 03:26:39.538826 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:26:39.541908 (+ 3082us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:26:39.544255 (+ 2347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:26:39.544260 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:26:39.544263 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:26:39.544423 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:26:39.545493 (+ 1070us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:26:39.545496 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:26:39.545498 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:26:39.545801 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:26:39.545977 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0810 03:26:39.547070 (+ 1093us) server_negotiation.cc:299] Negotiation successful
0810 03:26:39.547317 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11481,"spinlock_wait_cycles":7214464,"thread_start_us":11683,"threads_started":1}
I20260810 03:26:39.550101 7056 heartbeater.cc:347] Connected to a master server at 127.27.161.61:43223
I20260810 03:26:39.550266 7056 heartbeater.cc:464] Registering TS with master...
I20260810 03:26:39.550818 7056 heartbeater.cc:511] Master 127.27.161.61:43223 requested a full tablet report, sending...
I20260810 03:26:39.551781 6069 ts_manager.cc:194] Registered new tserver with Master: 05f07b3f00144d75b6c6fc9a40632f17 (127.27.161.3:45853)
I20260810 03:26:39.551045 5158 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37096:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 03:26:39.564198 5158 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 03:26:39.633741 7095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:39.603376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57854 (local address 127.27.161.3:45853)
0810 03:26:39.612626 (+ 9250us) server_negotiation.cc:206] Beginning negotiation
0810 03:26:39.612639 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 03:26:39.612692 (+ 53us) server_negotiation.cc:407] Connection header received
0810 03:26:39.612832 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:26:39.612843 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:26:39.613190 (+ 347us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:26:39.613491 (+ 301us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:26:39.616003 (+ 2512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:39.621078 (+ 5075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:39.625915 (+ 4837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:39.628118 (+ 2203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:39.630788 (+ 2670us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:39.631248 (+ 460us) server_negotiation.cc:1035] Waiting for connection context
0810 03:26:39.631695 (+ 447us) server_negotiation.cc:299] Negotiation successful
0810 03:26:39.632525 (+ 830us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8994,"thread_start_us":8769,"threads_started":1}
I20260810 03:26:39.634455 7094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:39.603074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.3:45853 (local address 127.0.0.1:57854)
0810 03:26:39.604052 (+ 978us) negotiation.cc:107] Waiting for socket to connect
0810 03:26:39.604072 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 03:26:39.604214 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:26:39.613492 (+ 9278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:26:39.613499 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:26:39.613536 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:26:39.614224 (+ 688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:26:39.614245 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:26:39.621338 (+ 7093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:26:39.621346 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:26:39.625573 (+ 4227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:26:39.625592 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:26:39.630871 (+ 5279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:26:39.630877 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:26:39.631225 (+ 348us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:39.631264 (+ 39us) client_negotiation.cc:769] Sending connection context
0810 03:26:39.631479 (+ 215us) client_negotiation.cc:240] Negotiation successful
0810 03:26:39.632210 (+ 731us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":831,"thread_start_us":621,"threads_started":1}
I20260810 03:26:39.639518 7096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:39.605075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.1:36623 (local address 127.0.0.1:55058)
0810 03:26:39.624641 (+ 19566us) negotiation.cc:107] Waiting for socket to connect
0810 03:26:39.624662 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0810 03:26:39.624805 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:26:39.625897 (+ 1092us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:26:39.625902 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:26:39.625933 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:26:39.626368 (+ 435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:26:39.626382 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:26:39.630379 (+ 3997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:26:39.630411 (+ 32us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:26:39.635765 (+ 5354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:26:39.635786 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:26:39.638518 (+ 2732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:26:39.638537 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:26:39.638807 (+ 270us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:39.638845 (+ 38us) client_negotiation.cc:769] Sending connection context
0810 03:26:39.638926 (+ 81us) client_negotiation.cc:240] Negotiation successful
0810 03:26:39.639078 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19370,"thread_start_us":17974,"threads_started":1}
I20260810 03:26:39.640578 7100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:39.605344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55058 (local address 127.27.161.1:36623)
0810 03:26:39.619910 (+ 14566us) server_negotiation.cc:206] Beginning negotiation
0810 03:26:39.619923 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 03:26:39.624823 (+ 4900us) server_negotiation.cc:407] Connection header received
0810 03:26:39.624970 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:26:39.624979 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:26:39.625325 (+ 346us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:26:39.625556 (+ 231us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:26:39.626688 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:39.629592 (+ 2904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:39.635985 (+ 6393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:39.637849 (+ 1864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:39.638081 (+ 232us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:39.638312 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0810 03:26:39.639910 (+ 1598us) server_negotiation.cc:299] Negotiation successful
0810 03:26:39.640264 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14294,"thread_start_us":14689,"threads_started":1}
I20260810 03:26:39.639930 6953 tablet_service.cc:1467] Processing CreateTablet for tablet 5924a9d42de94104a90d386d62dc5cad (DEFAULT_TABLE table=test-table [id=d061d3ebbdbc4ccc9e62350f030f0f22]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:26:39.641995 6953 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5924a9d42de94104a90d386d62dc5cad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:26:39.645361 7099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:39.604676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.2:43345 (local address 127.0.0.1:60408)
0810 03:26:39.623931 (+ 19255us) negotiation.cc:107] Waiting for socket to connect
0810 03:26:39.623958 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0810 03:26:39.624262 (+ 304us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:26:39.630340 (+ 6078us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:26:39.630374 (+ 34us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:26:39.630585 (+ 211us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:26:39.634463 (+ 3878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:26:39.634483 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:26:39.638972 (+ 4489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:26:39.638977 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:26:39.641428 (+ 2451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:26:39.641449 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:26:39.643888 (+ 2439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:26:39.643897 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:26:39.644190 (+ 293us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:39.644231 (+ 41us) client_negotiation.cc:769] Sending connection context
0810 03:26:39.644333 (+ 102us) client_negotiation.cc:240] Negotiation successful
0810 03:26:39.644974 (+ 641us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19084,"thread_start_us":18069,"threads_started":1}
I20260810 03:26:39.646189 7097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:39.604977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60408 (local address 127.27.161.2:43345)
0810 03:26:39.622282 (+ 17305us) server_negotiation.cc:206] Beginning negotiation
0810 03:26:39.622294 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 03:26:39.624989 (+ 2695us) server_negotiation.cc:407] Connection header received
0810 03:26:39.625130 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:26:39.625140 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:26:39.625441 (+ 301us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:26:39.625734 (+ 293us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:26:39.634750 (+ 9016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:39.638765 (+ 4015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:39.641655 (+ 2890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:39.643298 (+ 1643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:39.643450 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:39.643676 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0810 03:26:39.645675 (+ 1999us) server_negotiation.cc:299] Negotiation successful
0810 03:26:39.645863 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17074,"thread_start_us":16928,"threads_started":1}
I20260810 03:26:39.647096 6537 tablet_service.cc:1467] Processing CreateTablet for tablet 5924a9d42de94104a90d386d62dc5cad (DEFAULT_TABLE table=test-table [id=d061d3ebbdbc4ccc9e62350f030f0f22]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:26:39.648470 6537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5924a9d42de94104a90d386d62dc5cad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:26:39.659879 6742 tablet_service.cc:1467] Processing CreateTablet for tablet 5924a9d42de94104a90d386d62dc5cad (DEFAULT_TABLE table=test-table [id=d061d3ebbdbc4ccc9e62350f030f0f22]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:26:39.662338 6742 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5924a9d42de94104a90d386d62dc5cad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:26:39.685848 7102 tablet_bootstrap.cc:492] T 5924a9d42de94104a90d386d62dc5cad P b7227f1049a6433489fae09b53501439: Bootstrap starting.
I20260810 03:26:39.690960 7102 tablet_bootstrap.cc:654] T 5924a9d42de94104a90d386d62dc5cad P b7227f1049a6433489fae09b53501439: Neither blocks nor log segments found. Creating new log.
I20260810 03:26:39.691325 7103 tablet_bootstrap.cc:492] T 5924a9d42de94104a90d386d62dc5cad P 05f07b3f00144d75b6c6fc9a40632f17: Bootstrap starting.
I20260810 03:26:39.693318 7102 log.cc:824] T 5924a9d42de94104a90d386d62dc5cad P b7227f1049a6433489fae09b53501439: Log is configured to *not* fsync() on all Append() calls
I20260810 03:26:39.698580 7105 tablet_bootstrap.cc:492] T 5924a9d42de94104a90d386d62dc5cad P b851ca7ac5ca41af917e9e02fb356ce8: Bootstrap starting.
I20260810 03:26:39.701579 7103 tablet_bootstrap.cc:654] T 5924a9d42de94104a90d386d62dc5cad P 05f07b3f00144d75b6c6fc9a40632f17: Neither blocks nor log segments found. Creating new log.
I20260810 03:26:39.703781 7103 log.cc:824] T 5924a9d42de94104a90d386d62dc5cad P 05f07b3f00144d75b6c6fc9a40632f17: Log is configured to *not* fsync() on all Append() calls
I20260810 03:26:39.704898 7105 tablet_bootstrap.cc:654] T 5924a9d42de94104a90d386d62dc5cad P b851ca7ac5ca41af917e9e02fb356ce8: Neither blocks nor log segments found. Creating new log.
I20260810 03:26:39.705261 7102 tablet_bootstrap.cc:492] T 5924a9d42de94104a90d386d62dc5cad P b7227f1049a6433489fae09b53501439: No bootstrap required, opened a new log
I20260810 03:26:39.705626 7102 ts_tablet_manager.cc:1397] T 5924a9d42de94104a90d386d62dc5cad P b7227f1049a6433489fae09b53501439: Time spent bootstrapping tablet: real 0.020s user 0.008s sys 0.008s
I20260810 03:26:39.707209 7105 log.cc:824] T 5924a9d42de94104a90d386d62dc5cad P b851ca7ac5ca41af917e9e02fb356ce8: Log is configured to *not* fsync() on all Append() calls
I20260810 03:26:39.719864 7102 raft_consensus.cc:348] T 5924a9d42de94104a90d386d62dc5cad P b7227f1049a6433489fae09b53501439 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f07b3f00144d75b6c6fc9a40632f17" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 45853 } } peers { permanent_uuid: "b851ca7ac5ca41af917e9e02fb356ce8" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 43345 } } peers { permanent_uuid: "b7227f1049a6433489fae09b53501439" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 36623 } }
I20260810 03:26:39.721628 7102 raft_consensus.cc:374] T 5924a9d42de94104a90d386d62dc5cad P b7227f1049a6433489fae09b53501439 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:26:39.721866 7102 raft_consensus.cc:729] T 5924a9d42de94104a90d386d62dc5cad P b7227f1049a6433489fae09b53501439 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7227f1049a6433489fae09b53501439, State: Initialized, Role: FOLLOWER
I20260810 03:26:39.722368 7103 tablet_bootstrap.cc:492] T 5924a9d42de94104a90d386d62dc5cad P 05f07b3f00144d75b6c6fc9a40632f17: No bootstrap required, opened a new log
I20260810 03:26:39.723301 7103 ts_tablet_manager.cc:1397] T 5924a9d42de94104a90d386d62dc5cad P 05f07b3f00144d75b6c6fc9a40632f17: Time spent bootstrapping tablet: real 0.032s user 0.018s sys 0.007s
I20260810 03:26:39.723774 7102 consensus_queue.cc:260] T 5924a9d42de94104a90d386d62dc5cad P b7227f1049a6433489fae09b53501439 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f07b3f00144d75b6c6fc9a40632f17" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 45853 } } peers { permanent_uuid: "b851ca7ac5ca41af917e9e02fb356ce8" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 43345 } } peers { permanent_uuid: "b7227f1049a6433489fae09b53501439" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 36623 } }
I20260810 03:26:39.724619 7105 tablet_bootstrap.cc:492] T 5924a9d42de94104a90d386d62dc5cad P b851ca7ac5ca41af917e9e02fb356ce8: No bootstrap required, opened a new log
I20260810 03:26:39.725250 7105 ts_tablet_manager.cc:1397] T 5924a9d42de94104a90d386d62dc5cad P b851ca7ac5ca41af917e9e02fb356ce8: Time spent bootstrapping tablet: real 0.027s user 0.017s sys 0.004s
I20260810 03:26:39.740710 7102 ts_tablet_manager.cc:1428] T 5924a9d42de94104a90d386d62dc5cad P b7227f1049a6433489fae09b53501439: Time spent starting tablet: real 0.035s user 0.023s sys 0.007s
I20260810 03:26:39.740708 7105 raft_consensus.cc:348] T 5924a9d42de94104a90d386d62dc5cad P b851ca7ac5ca41af917e9e02fb356ce8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f07b3f00144d75b6c6fc9a40632f17" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 45853 } } peers { permanent_uuid: "b851ca7ac5ca41af917e9e02fb356ce8" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 43345 } } peers { permanent_uuid: "b7227f1049a6433489fae09b53501439" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 36623 } }
I20260810 03:26:39.742152 7105 raft_consensus.cc:374] T 5924a9d42de94104a90d386d62dc5cad P b851ca7ac5ca41af917e9e02fb356ce8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:26:39.742456 7105 raft_consensus.cc:729] T 5924a9d42de94104a90d386d62dc5cad P b851ca7ac5ca41af917e9e02fb356ce8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b851ca7ac5ca41af917e9e02fb356ce8, State: Initialized, Role: FOLLOWER
I20260810 03:26:39.746750 7103 raft_consensus.cc:348] T 5924a9d42de94104a90d386d62dc5cad P 05f07b3f00144d75b6c6fc9a40632f17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f07b3f00144d75b6c6fc9a40632f17" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 45853 } } peers { permanent_uuid: "b851ca7ac5ca41af917e9e02fb356ce8" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 43345 } } peers { permanent_uuid: "b7227f1049a6433489fae09b53501439" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 36623 } }
I20260810 03:26:39.747457 7103 raft_consensus.cc:374] T 5924a9d42de94104a90d386d62dc5cad P 05f07b3f00144d75b6c6fc9a40632f17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:26:39.747666 7103 raft_consensus.cc:729] T 5924a9d42de94104a90d386d62dc5cad P 05f07b3f00144d75b6c6fc9a40632f17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05f07b3f00144d75b6c6fc9a40632f17, State: Initialized, Role: FOLLOWER
I20260810 03:26:39.748503 7103 consensus_queue.cc:260] T 5924a9d42de94104a90d386d62dc5cad P 05f07b3f00144d75b6c6fc9a40632f17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f07b3f00144d75b6c6fc9a40632f17" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 45853 } } peers { permanent_uuid: "b851ca7ac5ca41af917e9e02fb356ce8" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 43345 } } peers { permanent_uuid: "b7227f1049a6433489fae09b53501439" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 36623 } }
I20260810 03:26:39.756194 7103 ts_tablet_manager.cc:1428] T 5924a9d42de94104a90d386d62dc5cad P 05f07b3f00144d75b6c6fc9a40632f17: Time spent starting tablet: real 0.033s user 0.019s sys 0.006s
I20260810 03:26:39.756852 7054 heartbeater.cc:503] Master 127.27.161.62:36281 was elected leader, sending a full tablet report...
I20260810 03:26:39.757824 7105 consensus_queue.cc:260] T 5924a9d42de94104a90d386d62dc5cad P b851ca7ac5ca41af917e9e02fb356ce8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f07b3f00144d75b6c6fc9a40632f17" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 45853 } } peers { permanent_uuid: "b851ca7ac5ca41af917e9e02fb356ce8" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 43345 } } peers { permanent_uuid: "b7227f1049a6433489fae09b53501439" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 36623 } }
I20260810 03:26:39.760406 7113 raft_consensus.cc:482] T 5924a9d42de94104a90d386d62dc5cad P b851ca7ac5ca41af917e9e02fb356ce8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 03:26:39.761009 7113 raft_consensus.cc:504] T 5924a9d42de94104a90d386d62dc5cad P b851ca7ac5ca41af917e9e02fb356ce8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f07b3f00144d75b6c6fc9a40632f17" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 45853 } } peers { permanent_uuid: "b851ca7ac5ca41af917e9e02fb356ce8" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 43345 } } peers { permanent_uuid: "b7227f1049a6433489fae09b53501439" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 36623 } }
I20260810 03:26:39.767093 7113 leader_election.cc:288] T 5924a9d42de94104a90d386d62dc5cad P b851ca7ac5ca41af917e9e02fb356ce8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 05f07b3f00144d75b6c6fc9a40632f17 (127.27.161.3:45853), b7227f1049a6433489fae09b53501439 (127.27.161.1:36623)
I20260810 03:26:39.768764 6844 heartbeater.cc:503] Master 127.27.161.62:36281 was elected leader, sending a full tablet report...
I20260810 03:26:39.772066 7105 ts_tablet_manager.cc:1428] T 5924a9d42de94104a90d386d62dc5cad P b851ca7ac5ca41af917e9e02fb356ce8: Time spent starting tablet: real 0.047s user 0.014s sys 0.021s
W20260810 03:26:39.779824 6633 tablet.cc:2369] T 5924a9d42de94104a90d386d62dc5cad P b7227f1049a6433489fae09b53501439: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 03:26:39.784124 6846 tablet.cc:2369] T 5924a9d42de94104a90d386d62dc5cad P b851ca7ac5ca41af917e9e02fb356ce8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 03:26:39.791320 7095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:39.767752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.161.2:47485 (local address 127.27.161.3:45853)
0810 03:26:39.768842 (+ 1090us) server_negotiation.cc:206] Beginning negotiation
0810 03:26:39.768850 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 03:26:39.770011 (+ 1161us) server_negotiation.cc:407] Connection header received
0810 03:26:39.770104 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:26:39.770110 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:26:39.770198 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:26:39.770296 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:26:39.771746 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:39.772940 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:39.788464 (+ 15524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:39.789916 (+ 1452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:39.790121 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:39.790213 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0810 03:26:39.790895 (+ 682us) server_negotiation.cc:299] Negotiation successful
0810 03:26:39.791044 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":905}
I20260810 03:26:39.795257 7119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:39.767466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.3:45853 (local address 127.27.161.2:47485)
0810 03:26:39.769730 (+ 2264us) negotiation.cc:107] Waiting for socket to connect
0810 03:26:39.769754 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0810 03:26:39.769924 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:26:39.770601 (+ 677us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:26:39.770606 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:26:39.770643 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:26:39.771258 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:26:39.771278 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:26:39.773120 (+ 1842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:26:39.773127 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:26:39.788136 (+ 15009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:26:39.788169 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:26:39.790471 (+ 2302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:26:39.790479 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:26:39.790688 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:39.790725 (+ 37us) client_negotiation.cc:769] Sending connection context
0810 03:26:39.791794 (+ 1069us) client_negotiation.cc:240] Negotiation successful
0810 03:26:39.791956 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2087,"thread_start_us":2154,"threads_started":1}
I20260810 03:26:39.796700 6981 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5924a9d42de94104a90d386d62dc5cad" candidate_uuid: "b851ca7ac5ca41af917e9e02fb356ce8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05f07b3f00144d75b6c6fc9a40632f17" is_pre_election: true
I20260810 03:26:39.797310 6981 raft_consensus.cc:2454] T 5924a9d42de94104a90d386d62dc5cad P 05f07b3f00144d75b6c6fc9a40632f17 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b851ca7ac5ca41af917e9e02fb356ce8 in term 0.
I20260810 03:26:39.798369 6674 leader_election.cc:302] T 5924a9d42de94104a90d386d62dc5cad P b851ca7ac5ca41af917e9e02fb356ce8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05f07b3f00144d75b6c6fc9a40632f17, b851ca7ac5ca41af917e9e02fb356ce8; no voters:
I20260810 03:26:39.798998 7117 raft_consensus.cc:2781] T 5924a9d42de94104a90d386d62dc5cad P b851ca7ac5ca41af917e9e02fb356ce8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 03:26:39.799180 7117 raft_consensus.cc:482] T 5924a9d42de94104a90d386d62dc5cad P b851ca7ac5ca41af917e9e02fb356ce8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 03:26:39.799365 7117 raft_consensus.cc:3037] T 5924a9d42de94104a90d386d62dc5cad P b851ca7ac5ca41af917e9e02fb356ce8 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:26:39.800257 7100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:39.769371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.161.2:56941 (local address 127.27.161.1:36623)
0810 03:26:39.769585 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0810 03:26:39.769592 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:26:39.789292 (+ 19700us) server_negotiation.cc:407] Connection header received
0810 03:26:39.789773 (+ 481us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:26:39.789781 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:26:39.789885 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:26:39.790007 (+ 122us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:26:39.792431 (+ 2424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:39.793582 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:39.797637 (+ 4055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:39.798867 (+ 1230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:39.799009 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:39.799088 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0810 03:26:39.799773 (+ 685us) server_negotiation.cc:299] Negotiation successful
0810 03:26:39.799923 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260810 03:26:39.800971 7120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:39.768353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.1:36623 (local address 127.27.161.2:56941)
0810 03:26:39.789143 (+ 20790us) negotiation.cc:107] Waiting for socket to connect
0810 03:26:39.789168 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0810 03:26:39.789494 (+ 326us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:26:39.790319 (+ 825us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:26:39.790326 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:26:39.790369 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:26:39.792204 (+ 1835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:26:39.792220 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:26:39.793769 (+ 1549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:26:39.793775 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:26:39.797417 (+ 3642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:26:39.797436 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:26:39.799337 (+ 1901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:26:39.799344 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:26:39.799556 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:39.799591 (+ 35us) client_negotiation.cc:769] Sending connection context
0810 03:26:39.800440 (+ 849us) client_negotiation.cc:240] Negotiation successful
0810 03:26:39.800617 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20554,"thread_start_us":23684,"threads_started":1}
I20260810 03:26:39.802239 6563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5924a9d42de94104a90d386d62dc5cad" candidate_uuid: "b851ca7ac5ca41af917e9e02fb356ce8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7227f1049a6433489fae09b53501439" is_pre_election: true
I20260810 03:26:39.802821 6563 raft_consensus.cc:2454] T 5924a9d42de94104a90d386d62dc5cad P b7227f1049a6433489fae09b53501439 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b851ca7ac5ca41af917e9e02fb356ce8 in term 0.
I20260810 03:26:39.806934 7117 raft_consensus.cc:504] T 5924a9d42de94104a90d386d62dc5cad P b851ca7ac5ca41af917e9e02fb356ce8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f07b3f00144d75b6c6fc9a40632f17" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 45853 } } peers { permanent_uuid: "b851ca7ac5ca41af917e9e02fb356ce8" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 43345 } } peers { permanent_uuid: "b7227f1049a6433489fae09b53501439" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 36623 } }
I20260810 03:26:39.807682 7117 leader_election.cc:288] T 5924a9d42de94104a90d386d62dc5cad P b851ca7ac5ca41af917e9e02fb356ce8 [CANDIDATE]: Term 1 election: Requested vote from peers 05f07b3f00144d75b6c6fc9a40632f17 (127.27.161.3:45853), b7227f1049a6433489fae09b53501439 (127.27.161.1:36623)
I20260810 03:26:39.808176 6563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5924a9d42de94104a90d386d62dc5cad" candidate_uuid: "b851ca7ac5ca41af917e9e02fb356ce8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7227f1049a6433489fae09b53501439"
I20260810 03:26:39.808446 6981 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5924a9d42de94104a90d386d62dc5cad" candidate_uuid: "b851ca7ac5ca41af917e9e02fb356ce8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05f07b3f00144d75b6c6fc9a40632f17"
I20260810 03:26:39.808488 6563 raft_consensus.cc:3037] T 5924a9d42de94104a90d386d62dc5cad P b7227f1049a6433489fae09b53501439 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:26:39.808714 6981 raft_consensus.cc:3037] T 5924a9d42de94104a90d386d62dc5cad P 05f07b3f00144d75b6c6fc9a40632f17 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:26:39.812145 6563 raft_consensus.cc:2454] T 5924a9d42de94104a90d386d62dc5cad P b7227f1049a6433489fae09b53501439 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b851ca7ac5ca41af917e9e02fb356ce8 in term 1.
I20260810 03:26:39.812680 6981 raft_consensus.cc:2454] T 5924a9d42de94104a90d386d62dc5cad P 05f07b3f00144d75b6c6fc9a40632f17 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b851ca7ac5ca41af917e9e02fb356ce8 in term 1.
I20260810 03:26:39.812856 6672 leader_election.cc:302] T 5924a9d42de94104a90d386d62dc5cad P b851ca7ac5ca41af917e9e02fb356ce8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b7227f1049a6433489fae09b53501439, b851ca7ac5ca41af917e9e02fb356ce8; no voters:
I20260810 03:26:39.813577 7117 raft_consensus.cc:2781] T 5924a9d42de94104a90d386d62dc5cad P b851ca7ac5ca41af917e9e02fb356ce8 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:26:39.813804 7117 raft_consensus.cc:686] T 5924a9d42de94104a90d386d62dc5cad P b851ca7ac5ca41af917e9e02fb356ce8 [term 1 LEADER]: Becoming Leader. State: Replica: b851ca7ac5ca41af917e9e02fb356ce8, State: Running, Role: LEADER
I20260810 03:26:39.814507 7117 consensus_queue.cc:237] T 5924a9d42de94104a90d386d62dc5cad P b851ca7ac5ca41af917e9e02fb356ce8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f07b3f00144d75b6c6fc9a40632f17" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 45853 } } peers { permanent_uuid: "b851ca7ac5ca41af917e9e02fb356ce8" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 43345 } } peers { permanent_uuid: "b7227f1049a6433489fae09b53501439" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 36623 } }
I20260810 03:26:39.820489 5158 catalog_manager.cc:5495] T 5924a9d42de94104a90d386d62dc5cad P b851ca7ac5ca41af917e9e02fb356ce8 reported cstate change: term changed from 0 to 1, leader changed from <none> to b851ca7ac5ca41af917e9e02fb356ce8 (127.27.161.2). New cstate: current_term: 1 leader_uuid: "b851ca7ac5ca41af917e9e02fb356ce8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f07b3f00144d75b6c6fc9a40632f17" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 45853 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b851ca7ac5ca41af917e9e02fb356ce8" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 43345 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b7227f1049a6433489fae09b53501439" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 36623 } health_report { overall_health: UNKNOWN } } }
I20260810 03:26:39.843508 7097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:39.837322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60418 (local address 127.27.161.2:43345)
0810 03:26:39.837549 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0810 03:26:39.837555 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:26:39.837634 (+ 79us) server_negotiation.cc:407] Connection header received
0810 03:26:39.837865 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:26:39.837869 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:26:39.838023 (+ 154us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:26:39.838165 (+ 142us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 03:26:39.838573 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:39.839351 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:39.840961 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:39.841537 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:39.841694 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:39.842236 (+ 542us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 03:26:39.842801 (+ 565us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 03:26:39.842938 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0810 03:26:39.843052 (+ 114us) server_negotiation.cc:299] Negotiation successful
0810 03:26:39.843143 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260810 03:26:39.853117 6981 raft_consensus.cc:1261] T 5924a9d42de94104a90d386d62dc5cad P 05f07b3f00144d75b6c6fc9a40632f17 [term 1 FOLLOWER]: Refusing update from remote peer b851ca7ac5ca41af917e9e02fb356ce8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:26:39.853112 6563 raft_consensus.cc:1261] T 5924a9d42de94104a90d386d62dc5cad P b7227f1049a6433489fae09b53501439 [term 1 FOLLOWER]: Refusing update from remote peer b851ca7ac5ca41af917e9e02fb356ce8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:26:39.854318 7117 consensus_queue.cc:1035] T 5924a9d42de94104a90d386d62dc5cad P b851ca7ac5ca41af917e9e02fb356ce8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "05f07b3f00144d75b6c6fc9a40632f17" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 45853 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:26:39.854693 7113 consensus_queue.cc:1035] T 5924a9d42de94104a90d386d62dc5cad P b851ca7ac5ca41af917e9e02fb356ce8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7227f1049a6433489fae09b53501439" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 36623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:26:39.880882 7127 mvcc.cc:205] Tried to move back new op lower bound from 7316817509787906048 to 7316817509645996032. Current Snapshot: MvccSnapshot[applied={T|T < 7316817509787906048}]
I20260810 03:26:39.896775 5158 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37096:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 03:26:39.897086 5158 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37096:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 03:26:39.906337 5158 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4b65187fc20346eda207dd58ee639ef2: Sending DeleteTablet for 3 replicas of tablet 5924a9d42de94104a90d386d62dc5cad
I20260810 03:26:39.908087 6537 tablet_service.cc:1514] Processing DeleteTablet for tablet 5924a9d42de94104a90d386d62dc5cad with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:26:39 UTC) from {username='oryx', principal='oryx/127.27.161.62@KRBTEST.COM'} at 127.0.0.1:55058
I20260810 03:26:39.908064 6742 tablet_service.cc:1514] Processing DeleteTablet for tablet 5924a9d42de94104a90d386d62dc5cad with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:26:39 UTC) from {username='oryx', principal='oryx/127.27.161.62@KRBTEST.COM'} at 127.0.0.1:60408
I20260810 03:26:39.908102 6953 tablet_service.cc:1514] Processing DeleteTablet for tablet 5924a9d42de94104a90d386d62dc5cad with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:26:39 UTC) from {username='oryx', principal='oryx/127.27.161.62@KRBTEST.COM'} at 127.0.0.1:57854
I20260810 03:26:39.920967 7135 tablet_replica.cc:330] T 5924a9d42de94104a90d386d62dc5cad P b7227f1049a6433489fae09b53501439: stopping tablet replica
I20260810 03:26:39.922147 7137 tablet_replica.cc:330] T 5924a9d42de94104a90d386d62dc5cad P b851ca7ac5ca41af917e9e02fb356ce8: stopping tablet replica
I20260810 03:26:39.922818 7135 raft_consensus.cc:2229] T 5924a9d42de94104a90d386d62dc5cad P b7227f1049a6433489fae09b53501439 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 03:26:39.923501 7137 raft_consensus.cc:2229] T 5924a9d42de94104a90d386d62dc5cad P b851ca7ac5ca41af917e9e02fb356ce8 [term 1 LEADER]: Raft consensus shutting down.
I20260810 03:26:39.924225 7135 raft_consensus.cc:2258] T 5924a9d42de94104a90d386d62dc5cad P b7227f1049a6433489fae09b53501439 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:26:39.925215 7137 raft_consensus.cc:2258] T 5924a9d42de94104a90d386d62dc5cad P b851ca7ac5ca41af917e9e02fb356ce8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:26:39.927846 7138 tablet_replica.cc:330] T 5924a9d42de94104a90d386d62dc5cad P 05f07b3f00144d75b6c6fc9a40632f17: stopping tablet replica
I20260810 03:26:39.928012 7135 ts_tablet_manager.cc:1905] T 5924a9d42de94104a90d386d62dc5cad P b7227f1049a6433489fae09b53501439: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:26:39.928853 7138 raft_consensus.cc:2229] T 5924a9d42de94104a90d386d62dc5cad P 05f07b3f00144d75b6c6fc9a40632f17 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 03:26:39.929436 7138 raft_consensus.cc:2258] T 5924a9d42de94104a90d386d62dc5cad P 05f07b3f00144d75b6c6fc9a40632f17 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:26:39.930163 7137 ts_tablet_manager.cc:1905] T 5924a9d42de94104a90d386d62dc5cad P b851ca7ac5ca41af917e9e02fb356ce8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:26:39.937083 28292 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iBH5/build/asan/bin/kudu pid 5113
I20260810 03:26:39.938270 7138 ts_tablet_manager.cc:1905] T 5924a9d42de94104a90d386d62dc5cad P 05f07b3f00144d75b6c6fc9a40632f17: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:26:39.938653 5203 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:26:39.940439 7135 ts_tablet_manager.cc:1918] T 5924a9d42de94104a90d386d62dc5cad P b7227f1049a6433489fae09b53501439: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 03:26:39.940748 7135 log.cc:1196] T 5924a9d42de94104a90d386d62dc5cad P b7227f1049a6433489fae09b53501439: Deleting WAL directory at /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786332336319544-28292-0/minicluster-data/ts-0/wal/wals/5924a9d42de94104a90d386d62dc5cad
I20260810 03:26:39.941495 7135 ts_tablet_manager.cc:1939] T 5924a9d42de94104a90d386d62dc5cad P b7227f1049a6433489fae09b53501439: Deleting consensus metadata
I20260810 03:26:39.945591 7137 ts_tablet_manager.cc:1918] T 5924a9d42de94104a90d386d62dc5cad P b851ca7ac5ca41af917e9e02fb356ce8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 03:26:39.945796 7137 log.cc:1196] T 5924a9d42de94104a90d386d62dc5cad P b851ca7ac5ca41af917e9e02fb356ce8: Deleting WAL directory at /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786332336319544-28292-0/minicluster-data/ts-1/wal/wals/5924a9d42de94104a90d386d62dc5cad
I20260810 03:26:39.946532 7137 ts_tablet_manager.cc:1939] T 5924a9d42de94104a90d386d62dc5cad P b851ca7ac5ca41af917e9e02fb356ce8: Deleting consensus metadata
I20260810 03:26:39.948441 7138 ts_tablet_manager.cc:1918] T 5924a9d42de94104a90d386d62dc5cad P 05f07b3f00144d75b6c6fc9a40632f17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 03:26:39.948690 7138 log.cc:1196] T 5924a9d42de94104a90d386d62dc5cad P 05f07b3f00144d75b6c6fc9a40632f17: Deleting WAL directory at /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786332336319544-28292-0/minicluster-data/ts-2/wal/wals/5924a9d42de94104a90d386d62dc5cad
I20260810 03:26:39.949357 7138 ts_tablet_manager.cc:1939] T 5924a9d42de94104a90d386d62dc5cad P 05f07b3f00144d75b6c6fc9a40632f17: Deleting consensus metadata
I20260810 03:26:39.955685 7070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:39.910982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50860 (local address 127.27.161.61:43223)
0810 03:26:39.911213 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0810 03:26:39.911219 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:26:39.927322 (+ 16103us) server_negotiation.cc:407] Connection header received
0810 03:26:39.927689 (+ 367us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:26:39.927695 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:26:39.927775 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:26:39.927834 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:26:39.931433 (+ 3599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:39.932599 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:39.934705 (+ 2106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:39.935389 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:39.935508 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:39.947527 (+ 12019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:26:39.947536 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:26:39.947541 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:26:39.947607 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:26:39.951977 (+ 4370us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:26:39.953084 (+ 1107us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:26:39.953090 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:26:39.953093 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:26:39.953363 (+ 270us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:26:39.953840 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:26:39.953845 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:26:39.953848 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:26:39.954279 (+ 431us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:26:39.954575 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0810 03:26:39.955073 (+ 498us) server_negotiation.cc:299] Negotiation successful
0810 03:26:39.955323 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260810 03:26:39.974009 7077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:39.935739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32962 (local address 127.27.161.60:43217)
0810 03:26:39.935976 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0810 03:26:39.935982 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:26:39.955854 (+ 19872us) server_negotiation.cc:407] Connection header received
0810 03:26:39.956173 (+ 319us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:26:39.956180 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:26:39.956281 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:26:39.956359 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:26:39.958662 (+ 2303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:39.959818 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:39.961897 (+ 2079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:39.962573 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:39.962696 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:39.966666 (+ 3970us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:26:39.966673 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:26:39.966679 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:26:39.966729 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:26:39.970641 (+ 3912us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:26:39.971548 (+ 907us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:26:39.971555 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:26:39.971559 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:26:39.971790 (+ 231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:26:39.972212 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:26:39.972216 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:26:39.972219 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:26:39.972662 (+ 443us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:26:39.972985 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0810 03:26:39.973291 (+ 306us) server_negotiation.cc:299] Negotiation successful
0810 03:26:39.973580 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260810 03:26:40.094282 5134 catalog_manager.cc:4841] TS b7227f1049a6433489fae09b53501439 (127.27.161.1:36623): tablet 5924a9d42de94104a90d386d62dc5cad (table test-table [id=d061d3ebbdbc4ccc9e62350f030f0f22]) successfully deleted
I20260810 03:26:40.094728 5139 catalog_manager.cc:4841] TS b851ca7ac5ca41af917e9e02fb356ce8 (127.27.161.2:43345): tablet 5924a9d42de94104a90d386d62dc5cad (table test-table [id=d061d3ebbdbc4ccc9e62350f030f0f22]) successfully deleted
W20260810 03:26:40.095217 5139 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.27.161.2:43345, user_credentials={real_user=slave}} blocked reactor thread for 146199us
I20260810 03:26:40.095942 28292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iBH5/build/asan/bin/kudu with pid 5113
I20260810 03:26:40.159242 7070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:40.142457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50870 (local address 127.27.161.61:43223)
0810 03:26:40.142653 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0810 03:26:40.142660 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:26:40.142681 (+ 21us) server_negotiation.cc:407] Connection header received
0810 03:26:40.142866 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:26:40.142871 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:26:40.142960 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:26:40.143063 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:26:40.144941 (+ 1878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:40.146112 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:40.147921 (+ 1809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:40.148342 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:40.148464 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:40.151651 (+ 3187us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:26:40.151656 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:26:40.151659 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:26:40.151696 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:26:40.155133 (+ 3437us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:26:40.155840 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:26:40.155846 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:26:40.155850 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:26:40.156015 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:26:40.157054 (+ 1039us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:26:40.157060 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:26:40.157063 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:26:40.157419 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:26:40.157619 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0810 03:26:40.158533 (+ 914us) server_negotiation.cc:299] Negotiation successful
0810 03:26:40.158768 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260810 03:26:40.163360 7077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:40.143456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32976 (local address 127.27.161.60:43217)
0810 03:26:40.143656 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0810 03:26:40.143662 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:26:40.144966 (+ 1304us) server_negotiation.cc:407] Connection header received
0810 03:26:40.145099 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:26:40.145103 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:26:40.145175 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:26:40.145259 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:26:40.147689 (+ 2430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:40.148742 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:40.150557 (+ 1815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:40.151169 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:40.151368 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:40.155121 (+ 3753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:26:40.155129 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:26:40.155133 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:26:40.155179 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:26:40.159289 (+ 4110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:26:40.161091 (+ 1802us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:26:40.161097 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:26:40.161100 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:26:40.161273 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:26:40.161686 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:26:40.161692 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:26:40.161695 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:26:40.162108 (+ 413us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:26:40.162359 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0810 03:26:40.162736 (+ 377us) server_negotiation.cc:299] Negotiation successful
0810 03:26:40.163001 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260810 03:26:40.889037 7151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:40.888161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:36281 (local address 127.27.161.2:44819)
0810 03:26:40.888654 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0810 03:26:40.888809 (+ 155us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.62:36281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":365,"threads_started":1}
W20260810 03:26:40.889525 6844 heartbeater.cc:644] Failed to heartbeat to 127.27.161.62:36281 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.27.161.62:36281: connect: Connection refused (error 111)
I20260810 03:26:40.890007 7154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:40.888818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:36281 (local address 127.27.161.1:42597)
0810 03:26:40.889574 (+ 756us) negotiation.cc:107] Waiting for socket to connect
0810 03:26:40.889743 (+ 169us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.62:36281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":611,"thread_start_us":500,"threads_started":1}
W20260810 03:26:40.890537 6631 heartbeater.cc:644] Failed to heartbeat to 127.27.161.62:36281 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.27.161.62:36281: connect: Connection refused (error 111)
I20260810 03:26:40.890611 7152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:40.888505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:36281 (local address 127.27.161.3:53243)
0810 03:26:40.890236 (+ 1731us) negotiation.cc:107] Waiting for socket to connect
0810 03:26:40.890394 (+ 158us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.62:36281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1598,"thread_start_us":1504,"threads_started":1}
W20260810 03:26:40.890918 7054 heartbeater.cc:644] Failed to heartbeat to 127.27.161.62:36281 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.27.161.62:36281: connect: Connection refused (error 111)
I20260810 03:26:41.405372 7156 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1245ea4d6f734899a3087452b0495ee2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4b65187fc20346eda207dd58ee639ef2)
I20260810 03:26:41.405625 7156 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1245ea4d6f734899a3087452b0495ee2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4b65187fc20346eda207dd58ee639ef2" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 36281 } } peers { permanent_uuid: "d102fadcc4b047ac9e652e1e0a4670a4" member_type: VOTER last_known_addr { host: "127.27.161.61" port: 43223 } attrs { promote: false } } peers { permanent_uuid: "1245ea4d6f734899a3087452b0495ee2" member_type: VOTER last_known_addr { host: "127.27.161.60" port: 43217 } attrs { promote: false } }
I20260810 03:26:41.406282 7159 raft_consensus.cc:482] T 00000000000000000000000000000000 P d102fadcc4b047ac9e652e1e0a4670a4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4b65187fc20346eda207dd58ee639ef2)
I20260810 03:26:41.406664 7156 leader_election.cc:288] T 00000000000000000000000000000000 P 1245ea4d6f734899a3087452b0495ee2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4b65187fc20346eda207dd58ee639ef2 (127.27.161.62:36281), d102fadcc4b047ac9e652e1e0a4670a4 (127.27.161.61:43223)
I20260810 03:26:41.406582 7159 raft_consensus.cc:504] T 00000000000000000000000000000000 P d102fadcc4b047ac9e652e1e0a4670a4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4b65187fc20346eda207dd58ee639ef2" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 36281 } } peers { permanent_uuid: "d102fadcc4b047ac9e652e1e0a4670a4" member_type: VOTER last_known_addr { host: "127.27.161.61" port: 43223 } attrs { promote: false } } peers { permanent_uuid: "1245ea4d6f734899a3087452b0495ee2" member_type: VOTER last_known_addr { host: "127.27.161.60" port: 43217 } attrs { promote: false } }
I20260810 03:26:41.407958 7162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:41.406912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:36281 (local address 127.0.0.1:37490)
0810 03:26:41.407429 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0810 03:26:41.407673 (+ 244us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.62:36281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":258,"threads_started":1}
I20260810 03:26:41.408360 7159 leader_election.cc:288] T 00000000000000000000000000000000 P d102fadcc4b047ac9e652e1e0a4670a4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4b65187fc20346eda207dd58ee639ef2 (127.27.161.62:36281), 1245ea4d6f734899a3087452b0495ee2 (127.27.161.60:43217)
W20260810 03:26:41.409024 6321 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.161.62:36281: connect: Connection refused (error 111)
I20260810 03:26:41.409804 7167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:41.408563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:36281 (local address 127.0.0.1:37494)
0810 03:26:41.409270 (+ 707us) negotiation.cc:107] Waiting for socket to connect
0810 03:26:41.409484 (+ 214us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.62:36281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":561,"thread_start_us":397,"threads_started":1}
W20260810 03:26:41.411732 6047 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.161.62:36281: connect: Connection refused (error 111)
I20260810 03:26:41.412041 7164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:41.411323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:36281 (local address 127.0.0.1:37498)
0810 03:26:41.411596 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0810 03:26:41.411798 (+ 202us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.62:36281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149}
W20260810 03:26:41.412269 6321 leader_election.cc:334] T 00000000000000000000000000000000 P 1245ea4d6f734899a3087452b0495ee2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4b65187fc20346eda207dd58ee639ef2 (127.27.161.62:36281): Network error: Client connection negotiation failed: client connection to 127.27.161.62:36281: connect: Connection refused (error 111)
I20260810 03:26:41.414155 7169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:41.413561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:36281 (local address 127.0.0.1:37500)
0810 03:26:41.413760 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0810 03:26:41.413909 (+ 149us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.62:36281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260810 03:26:41.414371 6047 leader_election.cc:334] T 00000000000000000000000000000000 P d102fadcc4b047ac9e652e1e0a4670a4 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4b65187fc20346eda207dd58ee639ef2 (127.27.161.62:36281): Network error: Client connection negotiation failed: client connection to 127.27.161.62:36281: connect: Connection refused (error 111)
I20260810 03:26:41.416390 7162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:41.407577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.61:43223 (local address 127.0.0.1:50874)
0810 03:26:41.408279 (+ 702us) negotiation.cc:107] Waiting for socket to connect
0810 03:26:41.408297 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 03:26:41.408429 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:26:41.409358 (+ 929us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:26:41.409364 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:26:41.409394 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:26:41.409711 (+ 317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:26:41.409725 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:26:41.411744 (+ 2019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:26:41.411750 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:26:41.413911 (+ 2161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:26:41.413927 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:26:41.415677 (+ 1750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:26:41.415684 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:26:41.415863 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:41.415906 (+ 43us) client_negotiation.cc:769] Sending connection context
0810 03:26:41.416020 (+ 114us) client_negotiation.cc:240] Negotiation successful
0810 03:26:41.416167 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":612,"thread_start_us":1121,"threads_started":1}
I20260810 03:26:41.416479 7166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:41.407722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50874 (local address 127.27.161.61:43223)
0810 03:26:41.408979 (+ 1257us) server_negotiation.cc:206] Beginning negotiation
0810 03:26:41.408986 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:26:41.409006 (+ 20us) server_negotiation.cc:407] Connection header received
0810 03:26:41.409086 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:26:41.409090 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:26:41.409193 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:26:41.409856 (+ 663us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:26:41.410110 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:41.411536 (+ 1426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:41.414099 (+ 2563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:41.415474 (+ 1375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:41.415609 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:41.415689 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0810 03:26:41.416044 (+ 355us) server_negotiation.cc:299] Negotiation successful
0810 03:26:41.416135 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1121,"thread_start_us":959,"threads_started":1}
I20260810 03:26:41.417021 6081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1245ea4d6f734899a3087452b0495ee2" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "d102fadcc4b047ac9e652e1e0a4670a4" is_pre_election: true
I20260810 03:26:41.417286 6081 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d102fadcc4b047ac9e652e1e0a4670a4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1245ea4d6f734899a3087452b0495ee2 in term 1.
I20260810 03:26:41.417793 6321 leader_election.cc:302] T 00000000000000000000000000000000 P 1245ea4d6f734899a3087452b0495ee2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1245ea4d6f734899a3087452b0495ee2, d102fadcc4b047ac9e652e1e0a4670a4; no voters: 4b65187fc20346eda207dd58ee639ef2
I20260810 03:26:41.418203 7156 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1245ea4d6f734899a3087452b0495ee2 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260810 03:26:41.418344 7156 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1245ea4d6f734899a3087452b0495ee2 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4b65187fc20346eda207dd58ee639ef2)
I20260810 03:26:41.418380 7167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:41.409335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.60:43217 (local address 127.0.0.1:32992)
0810 03:26:41.410084 (+ 749us) negotiation.cc:107] Waiting for socket to connect
0810 03:26:41.410102 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 03:26:41.410263 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:26:41.412231 (+ 1968us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:26:41.412235 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:26:41.412281 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:26:41.412604 (+ 323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:26:41.412617 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:26:41.414239 (+ 1622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:26:41.414249 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:26:41.416063 (+ 1814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:26:41.416078 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:26:41.417493 (+ 1415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:26:41.417499 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:26:41.417706 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:41.417737 (+ 31us) client_negotiation.cc:769] Sending connection context
0810 03:26:41.417844 (+ 107us) client_negotiation.cc:240] Negotiation successful
0810 03:26:41.418088 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":643,"mutex_wait_us":47,"thread_start_us":2068,"threads_started":1}
I20260810 03:26:41.418445 7156 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1245ea4d6f734899a3087452b0495ee2 [term 1 FOLLOWER]: Advancing to term 2
I20260810 03:26:41.418486 7170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:41.410040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32992 (local address 127.27.161.60:43217)
0810 03:26:41.411865 (+ 1825us) server_negotiation.cc:206] Beginning negotiation
0810 03:26:41.411873 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 03:26:41.411896 (+ 23us) server_negotiation.cc:407] Connection header received
0810 03:26:41.411964 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:26:41.411968 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:26:41.412040 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:26:41.412756 (+ 716us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:26:41.413020 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:41.414089 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:41.416222 (+ 2133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:41.417283 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:41.417392 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:41.417553 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0810 03:26:41.417868 (+ 315us) server_negotiation.cc:299] Negotiation successful
0810 03:26:41.418070 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1002,"thread_start_us":847,"threads_started":1}
I20260810 03:26:41.419195 6359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d102fadcc4b047ac9e652e1e0a4670a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "1245ea4d6f734899a3087452b0495ee2" is_pre_election: true
I20260810 03:26:41.421201 7156 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1245ea4d6f734899a3087452b0495ee2 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4b65187fc20346eda207dd58ee639ef2" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 36281 } } peers { permanent_uuid: "d102fadcc4b047ac9e652e1e0a4670a4" member_type: VOTER last_known_addr { host: "127.27.161.61" port: 43223 } attrs { promote: false } } peers { permanent_uuid: "1245ea4d6f734899a3087452b0495ee2" member_type: VOTER last_known_addr { host: "127.27.161.60" port: 43217 } attrs { promote: false } }
I20260810 03:26:41.421854 6359 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 1245ea4d6f734899a3087452b0495ee2 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d102fadcc4b047ac9e652e1e0a4670a4 in current term 2: Already voted for candidate 1245ea4d6f734899a3087452b0495ee2 in this term.
I20260810 03:26:41.422992 7162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:41.422555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:36281 (local address 127.0.0.1:37520)
0810 03:26:41.422728 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0810 03:26:41.422816 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.62:36281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260810 03:26:41.423869 6047 leader_election.cc:302] T 00000000000000000000000000000000 P d102fadcc4b047ac9e652e1e0a4670a4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d102fadcc4b047ac9e652e1e0a4670a4; no voters: 1245ea4d6f734899a3087452b0495ee2, 4b65187fc20346eda207dd58ee639ef2
I20260810 03:26:41.424244 7159 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d102fadcc4b047ac9e652e1e0a4670a4 [term 1 FOLLOWER]: Advancing to term 2
I20260810 03:26:41.424436 7162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:41.424026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:36281 (local address 127.0.0.1:37524)
0810 03:26:41.424217 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0810 03:26:41.424305 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.62:36281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
W20260810 03:26:41.424618 6321 leader_election.cc:334] T 00000000000000000000000000000000 P 1245ea4d6f734899a3087452b0495ee2 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 4b65187fc20346eda207dd58ee639ef2 (127.27.161.62:36281): Network error: Client connection negotiation failed: client connection to 127.27.161.62:36281: connect: Connection refused (error 111)
I20260810 03:26:41.425379 7156 leader_election.cc:288] T 00000000000000000000000000000000 P 1245ea4d6f734899a3087452b0495ee2 [CANDIDATE]: Term 2 election: Requested vote from peers 4b65187fc20346eda207dd58ee639ef2 (127.27.161.62:36281), d102fadcc4b047ac9e652e1e0a4670a4 (127.27.161.61:43223)
I20260810 03:26:41.425839 6081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1245ea4d6f734899a3087452b0495ee2" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "d102fadcc4b047ac9e652e1e0a4670a4"
I20260810 03:26:41.426783 7159 raft_consensus.cc:2726] T 00000000000000000000000000000000 P d102fadcc4b047ac9e652e1e0a4670a4 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 03:26:41.428974 6081 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d102fadcc4b047ac9e652e1e0a4670a4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1245ea4d6f734899a3087452b0495ee2 in term 2.
I20260810 03:26:41.429621 6321 leader_election.cc:302] T 00000000000000000000000000000000 P 1245ea4d6f734899a3087452b0495ee2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1245ea4d6f734899a3087452b0495ee2, d102fadcc4b047ac9e652e1e0a4670a4; no voters: 4b65187fc20346eda207dd58ee639ef2
I20260810 03:26:41.429888 7156 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1245ea4d6f734899a3087452b0495ee2 [term 2 FOLLOWER]: Leader election won for term 2
I20260810 03:26:41.430334 7156 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1245ea4d6f734899a3087452b0495ee2 [term 2 LEADER]: Becoming Leader. State: Replica: 1245ea4d6f734899a3087452b0495ee2, State: Running, Role: LEADER
I20260810 03:26:41.430701 7156 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1245ea4d6f734899a3087452b0495ee2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4b65187fc20346eda207dd58ee639ef2" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 36281 } } peers { permanent_uuid: "d102fadcc4b047ac9e652e1e0a4670a4" member_type: VOTER last_known_addr { host: "127.27.161.61" port: 43223 } attrs { promote: false } } peers { permanent_uuid: "1245ea4d6f734899a3087452b0495ee2" member_type: VOTER last_known_addr { host: "127.27.161.60" port: 43217 } attrs { promote: false } }
I20260810 03:26:41.432587 7172 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1245ea4d6f734899a3087452b0495ee2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1245ea4d6f734899a3087452b0495ee2. Latest consensus state: current_term: 2 leader_uuid: "1245ea4d6f734899a3087452b0495ee2" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4b65187fc20346eda207dd58ee639ef2" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 36281 } } peers { permanent_uuid: "d102fadcc4b047ac9e652e1e0a4670a4" member_type: VOTER last_known_addr { host: "127.27.161.61" port: 43223 } attrs { promote: false } } peers { permanent_uuid: "1245ea4d6f734899a3087452b0495ee2" member_type: VOTER last_known_addr { host: "127.27.161.60" port: 43217 } attrs { promote: false } } }
I20260810 03:26:41.433025 7172 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1245ea4d6f734899a3087452b0495ee2 [sys.catalog]: This master's current role is: LEADER
I20260810 03:26:41.433718 7175 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 03:26:41.438028 7175 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 03:26:41.438856 7175 catalog_manager.cc:1259] Loaded cluster ID: 75ff020955394fb1943d772cebcad889
I20260810 03:26:41.438933 7175 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 03:26:41.440181 7175 catalog_manager.cc:1504] Loading token signing keys...
I20260810 03:26:41.440960 7175 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 1245ea4d6f734899a3087452b0495ee2: Loaded TSK: 0
I20260810 03:26:41.441901 7175 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 03:26:41.480271 6346 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32976:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 03:26:41.488900 6346 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 03:26:41.494573 7162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:41.494072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:36281 (local address 127.0.0.1:37548)
0810 03:26:41.494264 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0810 03:26:41.494383 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.62:36281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260810 03:26:41.494601 6081 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d102fadcc4b047ac9e652e1e0a4670a4 [term 2 FOLLOWER]: Refusing update from remote peer 1245ea4d6f734899a3087452b0495ee2: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260810 03:26:41.495668 7162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:41.495303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:36281 (local address 127.0.0.1:37558)
0810 03:26:41.495431 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0810 03:26:41.495515 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.62:36281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260810 03:26:41.495750 7172 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1245ea4d6f734899a3087452b0495ee2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d102fadcc4b047ac9e652e1e0a4670a4" member_type: VOTER last_known_addr { host: "127.27.161.61" port: 43223 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260810 03:26:41.496023 6321 consensus_peers.cc:487] T 00000000000000000000000000000000 P 1245ea4d6f734899a3087452b0495ee2 -> Peer 4b65187fc20346eda207dd58ee639ef2 (127.27.161.62:36281): Couldn't send request to peer 4b65187fc20346eda207dd58ee639ef2. Status: Network error: Client connection negotiation failed: client connection to 127.27.161.62:36281: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260810 03:26:41.499200 7159 sys_catalog.cc:455] T 00000000000000000000000000000000 P d102fadcc4b047ac9e652e1e0a4670a4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1245ea4d6f734899a3087452b0495ee2. Latest consensus state: current_term: 2 leader_uuid: "1245ea4d6f734899a3087452b0495ee2" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4b65187fc20346eda207dd58ee639ef2" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 36281 } } peers { permanent_uuid: "d102fadcc4b047ac9e652e1e0a4670a4" member_type: VOTER last_known_addr { host: "127.27.161.61" port: 43223 } attrs { promote: false } } peers { permanent_uuid: "1245ea4d6f734899a3087452b0495ee2" member_type: VOTER last_known_addr { host: "127.27.161.60" port: 43217 } attrs { promote: false } } }
I20260810 03:26:41.499579 7159 sys_catalog.cc:458] T 00000000000000000000000000000000 P d102fadcc4b047ac9e652e1e0a4670a4 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 03:26:41.502261 7156 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1245ea4d6f734899a3087452b0495ee2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "1245ea4d6f734899a3087452b0495ee2" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4b65187fc20346eda207dd58ee639ef2" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 36281 } } peers { permanent_uuid: "d102fadcc4b047ac9e652e1e0a4670a4" member_type: VOTER last_known_addr { host: "127.27.161.61" port: 43223 } attrs { promote: false } } peers { permanent_uuid: "1245ea4d6f734899a3087452b0495ee2" member_type: VOTER last_known_addr { host: "127.27.161.60" port: 43217 } attrs { promote: false } } }
I20260810 03:26:41.502462 7159 sys_catalog.cc:455] T 00000000000000000000000000000000 P d102fadcc4b047ac9e652e1e0a4670a4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "1245ea4d6f734899a3087452b0495ee2" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4b65187fc20346eda207dd58ee639ef2" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 36281 } } peers { permanent_uuid: "d102fadcc4b047ac9e652e1e0a4670a4" member_type: VOTER last_known_addr { host: "127.27.161.61" port: 43223 } attrs { promote: false } } peers { permanent_uuid: "1245ea4d6f734899a3087452b0495ee2" member_type: VOTER last_known_addr { host: "127.27.161.60" port: 43217 } attrs { promote: false } } }
I20260810 03:26:41.502835 7156 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1245ea4d6f734899a3087452b0495ee2 [sys.catalog]: This master's current role is: LEADER
I20260810 03:26:41.502924 7159 sys_catalog.cc:458] T 00000000000000000000000000000000 P d102fadcc4b047ac9e652e1e0a4670a4 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 03:26:41.527994 7192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:41.514815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55062 (local address 127.27.161.1:36623)
0810 03:26:41.520467 (+ 5652us) server_negotiation.cc:206] Beginning negotiation
0810 03:26:41.520475 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 03:26:41.520498 (+ 23us) server_negotiation.cc:407] Connection header received
0810 03:26:41.520578 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:26:41.520583 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:26:41.520652 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:26:41.520836 (+ 184us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:26:41.521475 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:41.522745 (+ 1270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:41.525526 (+ 2781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:41.526923 (+ 1397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:41.527065 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:41.527203 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0810 03:26:41.527455 (+ 252us) server_negotiation.cc:299] Negotiation successful
0810 03:26:41.527609 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5423,"thread_start_us":5309,"threads_started":1}
I20260810 03:26:41.528054 7164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:41.514059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.1:36623 (local address 127.0.0.1:55062)
0810 03:26:41.514381 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0810 03:26:41.514406 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0810 03:26:41.514566 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:26:41.520870 (+ 6304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:26:41.520876 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:26:41.520924 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:26:41.521279 (+ 355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:26:41.521293 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:26:41.522985 (+ 1692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:26:41.522991 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:26:41.525355 (+ 2364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:26:41.525372 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:26:41.527111 (+ 1739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:26:41.527117 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:26:41.527285 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:41.527319 (+ 34us) client_negotiation.cc:769] Sending connection context
0810 03:26:41.527403 (+ 84us) client_negotiation.cc:240] Negotiation successful
0810 03:26:41.527564 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":198,"mutex_wait_us":257}
I20260810 03:26:41.528517 7194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:41.514904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60430 (local address 127.27.161.2:43345)
0810 03:26:41.520833 (+ 5929us) server_negotiation.cc:206] Beginning negotiation
0810 03:26:41.520842 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 03:26:41.520866 (+ 24us) server_negotiation.cc:407] Connection header received
0810 03:26:41.520960 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:26:41.520966 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:26:41.521043 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:26:41.522013 (+ 970us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:26:41.522201 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:41.523256 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:41.526344 (+ 3088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:41.527428 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:41.527555 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:41.527659 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0810 03:26:41.528067 (+ 408us) server_negotiation.cc:299] Negotiation successful
0810 03:26:41.528222 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5694,"thread_start_us":5536,"threads_started":1}
I20260810 03:26:41.528543 7187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:41.514527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.2:43345 (local address 127.0.0.1:60430)
0810 03:26:41.515207 (+ 680us) negotiation.cc:107] Waiting for socket to connect
0810 03:26:41.515224 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 03:26:41.515344 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:26:41.521226 (+ 5882us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:26:41.521231 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:26:41.521260 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:26:41.521774 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:26:41.521800 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:26:41.523413 (+ 1613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:26:41.523417 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:26:41.526092 (+ 2675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:26:41.526105 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:26:41.527701 (+ 1596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:26:41.527709 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:26:41.527874 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:41.527912 (+ 38us) client_negotiation.cc:769] Sending connection context
0810 03:26:41.528004 (+ 92us) client_negotiation.cc:240] Negotiation successful
0810 03:26:41.528143 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":561,"mutex_wait_us":25,"thread_start_us":567,"threads_started":1}
I20260810 03:26:41.529294 6537 tablet_service.cc:1467] Processing CreateTablet for tablet e6a73ce006274facbf8e092fd626fcd5 (DEFAULT_TABLE table=test-table [id=973af5d4d4ed49948553c21c1e4791ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:26:41.529774 6537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6a73ce006274facbf8e092fd626fcd5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:26:41.530539 7185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:41.513685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57858 (local address 127.27.161.3:45853)
0810 03:26:41.521410 (+ 7725us) server_negotiation.cc:206] Beginning negotiation
0810 03:26:41.521416 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:26:41.521433 (+ 17us) server_negotiation.cc:407] Connection header received
0810 03:26:41.521501 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:26:41.521504 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:26:41.521575 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:26:41.521833 (+ 258us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:26:41.522318 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:41.523207 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:41.526085 (+ 2878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:41.529577 (+ 3492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:41.529704 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:41.529815 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0810 03:26:41.530180 (+ 365us) server_negotiation.cc:299] Negotiation successful
0810 03:26:41.530327 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7590,"thread_start_us":7484,"threads_started":1}
I20260810 03:26:41.530596 7162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:41.513417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.3:45853 (local address 127.0.0.1:57858)
0810 03:26:41.513888 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0810 03:26:41.513910 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0810 03:26:41.514147 (+ 237us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:26:41.521884 (+ 7737us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:26:41.521889 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:26:41.521913 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:26:41.522176 (+ 263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:26:41.522188 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:26:41.523388 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:26:41.523394 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:26:41.525893 (+ 2499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:26:41.525911 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:26:41.529719 (+ 3808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:26:41.529725 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:26:41.529899 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:41.529934 (+ 35us) client_negotiation.cc:769] Sending connection context
0810 03:26:41.530059 (+ 125us) client_negotiation.cc:240] Negotiation successful
0810 03:26:41.530292 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"mutex_wait_us":47}
I20260810 03:26:41.534885 6953 tablet_service.cc:1467] Processing CreateTablet for tablet e6a73ce006274facbf8e092fd626fcd5 (DEFAULT_TABLE table=test-table [id=973af5d4d4ed49948553c21c1e4791ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:26:41.534891 6742 tablet_service.cc:1467] Processing CreateTablet for tablet e6a73ce006274facbf8e092fd626fcd5 (DEFAULT_TABLE table=test-table [id=973af5d4d4ed49948553c21c1e4791ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:26:41.535426 6953 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6a73ce006274facbf8e092fd626fcd5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:26:41.535439 6742 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6a73ce006274facbf8e092fd626fcd5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:26:41.538658 7195 tablet_bootstrap.cc:492] T e6a73ce006274facbf8e092fd626fcd5 P b7227f1049a6433489fae09b53501439: Bootstrap starting.
I20260810 03:26:41.540736 7195 tablet_bootstrap.cc:654] T e6a73ce006274facbf8e092fd626fcd5 P b7227f1049a6433489fae09b53501439: Neither blocks nor log segments found. Creating new log.
I20260810 03:26:41.543123 7198 tablet_bootstrap.cc:492] T e6a73ce006274facbf8e092fd626fcd5 P 05f07b3f00144d75b6c6fc9a40632f17: Bootstrap starting.
I20260810 03:26:41.544076 7195 tablet_bootstrap.cc:492] T e6a73ce006274facbf8e092fd626fcd5 P b7227f1049a6433489fae09b53501439: No bootstrap required, opened a new log
I20260810 03:26:41.544271 7195 ts_tablet_manager.cc:1397] T e6a73ce006274facbf8e092fd626fcd5 P b7227f1049a6433489fae09b53501439: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260810 03:26:41.544777 7198 tablet_bootstrap.cc:654] T e6a73ce006274facbf8e092fd626fcd5 P 05f07b3f00144d75b6c6fc9a40632f17: Neither blocks nor log segments found. Creating new log.
I20260810 03:26:41.545312 7195 raft_consensus.cc:348] T e6a73ce006274facbf8e092fd626fcd5 P b7227f1049a6433489fae09b53501439 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f07b3f00144d75b6c6fc9a40632f17" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 45853 } } peers { permanent_uuid: "b7227f1049a6433489fae09b53501439" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 36623 } } peers { permanent_uuid: "b851ca7ac5ca41af917e9e02fb356ce8" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 43345 } }
I20260810 03:26:41.545619 7195 raft_consensus.cc:374] T e6a73ce006274facbf8e092fd626fcd5 P b7227f1049a6433489fae09b53501439 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:26:41.545717 7195 raft_consensus.cc:729] T e6a73ce006274facbf8e092fd626fcd5 P b7227f1049a6433489fae09b53501439 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7227f1049a6433489fae09b53501439, State: Initialized, Role: FOLLOWER
I20260810 03:26:41.546094 7195 consensus_queue.cc:260] T e6a73ce006274facbf8e092fd626fcd5 P b7227f1049a6433489fae09b53501439 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f07b3f00144d75b6c6fc9a40632f17" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 45853 } } peers { permanent_uuid: "b7227f1049a6433489fae09b53501439" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 36623 } } peers { permanent_uuid: "b851ca7ac5ca41af917e9e02fb356ce8" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 43345 } }
I20260810 03:26:41.546409 7199 tablet_bootstrap.cc:492] T e6a73ce006274facbf8e092fd626fcd5 P b851ca7ac5ca41af917e9e02fb356ce8: Bootstrap starting.
I20260810 03:26:41.546998 7198 tablet_bootstrap.cc:492] T e6a73ce006274facbf8e092fd626fcd5 P 05f07b3f00144d75b6c6fc9a40632f17: No bootstrap required, opened a new log
I20260810 03:26:41.547165 7198 ts_tablet_manager.cc:1397] T e6a73ce006274facbf8e092fd626fcd5 P 05f07b3f00144d75b6c6fc9a40632f17: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20260810 03:26:41.548019 7198 raft_consensus.cc:348] T e6a73ce006274facbf8e092fd626fcd5 P 05f07b3f00144d75b6c6fc9a40632f17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f07b3f00144d75b6c6fc9a40632f17" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 45853 } } peers { permanent_uuid: "b7227f1049a6433489fae09b53501439" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 36623 } } peers { permanent_uuid: "b851ca7ac5ca41af917e9e02fb356ce8" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 43345 } }
I20260810 03:26:41.548272 7198 raft_consensus.cc:374] T e6a73ce006274facbf8e092fd626fcd5 P 05f07b3f00144d75b6c6fc9a40632f17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:26:41.548327 7195 ts_tablet_manager.cc:1428] T e6a73ce006274facbf8e092fd626fcd5 P b7227f1049a6433489fae09b53501439: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260810 03:26:41.548379 7198 raft_consensus.cc:729] T e6a73ce006274facbf8e092fd626fcd5 P 05f07b3f00144d75b6c6fc9a40632f17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05f07b3f00144d75b6c6fc9a40632f17, State: Initialized, Role: FOLLOWER
I20260810 03:26:41.548738 7199 tablet_bootstrap.cc:654] T e6a73ce006274facbf8e092fd626fcd5 P b851ca7ac5ca41af917e9e02fb356ce8: Neither blocks nor log segments found. Creating new log.
I20260810 03:26:41.548638 7198 consensus_queue.cc:260] T e6a73ce006274facbf8e092fd626fcd5 P 05f07b3f00144d75b6c6fc9a40632f17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f07b3f00144d75b6c6fc9a40632f17" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 45853 } } peers { permanent_uuid: "b7227f1049a6433489fae09b53501439" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 36623 } } peers { permanent_uuid: "b851ca7ac5ca41af917e9e02fb356ce8" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 43345 } }
I20260810 03:26:41.549711 7203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:41.547541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:36281 (local address 127.27.161.1:58743)
0810 03:26:41.549150 (+ 1609us) negotiation.cc:107] Waiting for socket to connect
0810 03:26:41.549385 (+ 235us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.62:36281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1453,"thread_start_us":1311,"threads_started":1}
I20260810 03:26:41.553155 7203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:41.551723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:36281 (local address 127.27.161.1:38679)
0810 03:26:41.552880 (+ 1157us) negotiation.cc:107] Waiting for socket to connect
0810 03:26:41.552984 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.62:36281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1034}
I20260810 03:26:41.553278 6630 heartbeater.cc:503] Master 127.27.161.60:43217 was elected leader, sending a full tablet report...
I20260810 03:26:41.556363 7198 ts_tablet_manager.cc:1428] T e6a73ce006274facbf8e092fd626fcd5 P 05f07b3f00144d75b6c6fc9a40632f17: Time spent starting tablet: real 0.009s user 0.002s sys 0.006s
I20260810 03:26:41.557540 7207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:41.556457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:36281 (local address 127.27.161.3:40665)
0810 03:26:41.557092 (+ 635us) negotiation.cc:107] Waiting for socket to connect
0810 03:26:41.557259 (+ 167us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.62:36281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":319,"threads_started":1}
I20260810 03:26:41.557554 7199 tablet_bootstrap.cc:492] T e6a73ce006274facbf8e092fd626fcd5 P b851ca7ac5ca41af917e9e02fb356ce8: No bootstrap required, opened a new log
I20260810 03:26:41.557854 7199 ts_tablet_manager.cc:1397] T e6a73ce006274facbf8e092fd626fcd5 P b851ca7ac5ca41af917e9e02fb356ce8: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.006s
I20260810 03:26:41.558595 7207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:41.558223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:36281 (local address 127.27.161.3:38113)
0810 03:26:41.558395 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0810 03:26:41.558471 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.62:36281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66,"threads_started":1}
I20260810 03:26:41.558897 7199 raft_consensus.cc:348] T e6a73ce006274facbf8e092fd626fcd5 P b851ca7ac5ca41af917e9e02fb356ce8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f07b3f00144d75b6c6fc9a40632f17" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 45853 } } peers { permanent_uuid: "b7227f1049a6433489fae09b53501439" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 36623 } } peers { permanent_uuid: "b851ca7ac5ca41af917e9e02fb356ce8" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 43345 } }
I20260810 03:26:41.559165 7199 raft_consensus.cc:374] T e6a73ce006274facbf8e092fd626fcd5 P b851ca7ac5ca41af917e9e02fb356ce8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:26:41.559276 7199 raft_consensus.cc:729] T e6a73ce006274facbf8e092fd626fcd5 P b851ca7ac5ca41af917e9e02fb356ce8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b851ca7ac5ca41af917e9e02fb356ce8, State: Initialized, Role: FOLLOWER
I20260810 03:26:41.559437 7053 heartbeater.cc:503] Master 127.27.161.60:43217 was elected leader, sending a full tablet report...
I20260810 03:26:41.559533 7199 consensus_queue.cc:260] T e6a73ce006274facbf8e092fd626fcd5 P b851ca7ac5ca41af917e9e02fb356ce8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f07b3f00144d75b6c6fc9a40632f17" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 45853 } } peers { permanent_uuid: "b7227f1049a6433489fae09b53501439" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 36623 } } peers { permanent_uuid: "b851ca7ac5ca41af917e9e02fb356ce8" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 43345 } }
I20260810 03:26:41.560438 7199 ts_tablet_manager.cc:1428] T e6a73ce006274facbf8e092fd626fcd5 P b851ca7ac5ca41af917e9e02fb356ce8: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20260810 03:26:41.563239 6843 heartbeater.cc:503] Master 127.27.161.60:43217 was elected leader, sending a full tablet report...
I20260810 03:26:41.563582 7212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:41.561429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:36281 (local address 127.27.161.2:39289)
0810 03:26:41.562953 (+ 1524us) negotiation.cc:107] Waiting for socket to connect
0810 03:26:41.563174 (+ 221us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.62:36281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1379,"thread_start_us":1227,"threads_started":1}
I20260810 03:26:41.739872 7204 raft_consensus.cc:482] T e6a73ce006274facbf8e092fd626fcd5 P 05f07b3f00144d75b6c6fc9a40632f17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 03:26:41.740154 7204 raft_consensus.cc:504] T e6a73ce006274facbf8e092fd626fcd5 P 05f07b3f00144d75b6c6fc9a40632f17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f07b3f00144d75b6c6fc9a40632f17" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 45853 } } peers { permanent_uuid: "b7227f1049a6433489fae09b53501439" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 36623 } } peers { permanent_uuid: "b851ca7ac5ca41af917e9e02fb356ce8" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 43345 } }
I20260810 03:26:41.741803 7204 leader_election.cc:288] T e6a73ce006274facbf8e092fd626fcd5 P 05f07b3f00144d75b6c6fc9a40632f17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b7227f1049a6433489fae09b53501439 (127.27.161.1:36623), b851ca7ac5ca41af917e9e02fb356ce8 (127.27.161.2:43345)
W20260810 03:26:41.748847 7060 tablet.cc:2369] T e6a73ce006274facbf8e092fd626fcd5 P 05f07b3f00144d75b6c6fc9a40632f17: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 03:26:41.750311 7194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:41.742399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.161.3:46717 (local address 127.27.161.2:43345)
0810 03:26:41.742617 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0810 03:26:41.742623 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:26:41.742977 (+ 354us) server_negotiation.cc:407] Connection header received
0810 03:26:41.743067 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:26:41.743072 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:26:41.743144 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:26:41.743241 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:26:41.743816 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:41.744968 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:41.747873 (+ 2905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:41.749347 (+ 1474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:41.749502 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:41.749603 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0810 03:26:41.749866 (+ 263us) server_negotiation.cc:299] Negotiation successful
0810 03:26:41.750041 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260810 03:26:41.750340 7207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:41.742362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.2:43345 (local address 127.27.161.3:46717)
0810 03:26:41.742718 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0810 03:26:41.742736 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 03:26:41.742886 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:26:41.743275 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:26:41.743279 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:26:41.743297 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:26:41.743638 (+ 341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:26:41.743652 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:26:41.745125 (+ 1473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:26:41.745131 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:26:41.747683 (+ 2552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:26:41.747696 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:26:41.749560 (+ 1864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:26:41.749566 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:26:41.749693 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:41.749720 (+ 27us) client_negotiation.cc:769] Sending connection context
0810 03:26:41.749809 (+ 89us) client_negotiation.cc:240] Negotiation successful
0810 03:26:41.749983 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":201,"mutex_wait_us":139}
I20260810 03:26:41.751991 7192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:41.742505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.161.3:56323 (local address 127.27.161.1:36623)
0810 03:26:41.742837 (+ 332us) server_negotiation.cc:206] Beginning negotiation
0810 03:26:41.742843 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:26:41.742878 (+ 35us) server_negotiation.cc:407] Connection header received
0810 03:26:41.742966 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:26:41.742971 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:26:41.743067 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:26:41.743170 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:26:41.743816 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:41.744977 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:41.749704 (+ 4727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:41.751042 (+ 1338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:41.751226 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:41.751301 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0810 03:26:41.751571 (+ 270us) server_negotiation.cc:299] Negotiation successful
0810 03:26:41.751712 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":177}
I20260810 03:26:41.752020 7210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:41.742259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.1:36623 (local address 127.27.161.3:56323)
0810 03:26:41.742613 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0810 03:26:41.742658 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0810 03:26:41.742831 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:26:41.743240 (+ 409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:26:41.743245 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:26:41.743277 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:26:41.743651 (+ 374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:26:41.743661 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:26:41.745125 (+ 1464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:26:41.745131 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:26:41.747679 (+ 2548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:26:41.747696 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:26:41.751243 (+ 3547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:26:41.751248 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:26:41.751381 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:41.751406 (+ 25us) client_negotiation.cc:769] Sending connection context
0810 03:26:41.751517 (+ 111us) client_negotiation.cc:240] Negotiation successful
0810 03:26:41.751660 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":221,"mutex_wait_us":101}
I20260810 03:26:41.752080 6775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6a73ce006274facbf8e092fd626fcd5" candidate_uuid: "05f07b3f00144d75b6c6fc9a40632f17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b851ca7ac5ca41af917e9e02fb356ce8" is_pre_election: true
I20260810 03:26:41.752466 6775 raft_consensus.cc:2454] T e6a73ce006274facbf8e092fd626fcd5 P b851ca7ac5ca41af917e9e02fb356ce8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 05f07b3f00144d75b6c6fc9a40632f17 in term 0.
I20260810 03:26:41.753033 6563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6a73ce006274facbf8e092fd626fcd5" candidate_uuid: "05f07b3f00144d75b6c6fc9a40632f17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7227f1049a6433489fae09b53501439" is_pre_election: true
I20260810 03:26:41.753167 6886 leader_election.cc:302] T e6a73ce006274facbf8e092fd626fcd5 P 05f07b3f00144d75b6c6fc9a40632f17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05f07b3f00144d75b6c6fc9a40632f17, b851ca7ac5ca41af917e9e02fb356ce8; no voters:
I20260810 03:26:41.753278 6563 raft_consensus.cc:2454] T e6a73ce006274facbf8e092fd626fcd5 P b7227f1049a6433489fae09b53501439 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 05f07b3f00144d75b6c6fc9a40632f17 in term 0.
I20260810 03:26:41.753643 7204 raft_consensus.cc:2781] T e6a73ce006274facbf8e092fd626fcd5 P 05f07b3f00144d75b6c6fc9a40632f17 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 03:26:41.753793 7204 raft_consensus.cc:482] T e6a73ce006274facbf8e092fd626fcd5 P 05f07b3f00144d75b6c6fc9a40632f17 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 03:26:41.753906 7204 raft_consensus.cc:3037] T e6a73ce006274facbf8e092fd626fcd5 P 05f07b3f00144d75b6c6fc9a40632f17 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:26:41.757392 7204 raft_consensus.cc:504] T e6a73ce006274facbf8e092fd626fcd5 P 05f07b3f00144d75b6c6fc9a40632f17 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f07b3f00144d75b6c6fc9a40632f17" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 45853 } } peers { permanent_uuid: "b7227f1049a6433489fae09b53501439" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 36623 } } peers { permanent_uuid: "b851ca7ac5ca41af917e9e02fb356ce8" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 43345 } }
I20260810 03:26:41.758147 7204 leader_election.cc:288] T e6a73ce006274facbf8e092fd626fcd5 P 05f07b3f00144d75b6c6fc9a40632f17 [CANDIDATE]: Term 1 election: Requested vote from peers b7227f1049a6433489fae09b53501439 (127.27.161.1:36623), b851ca7ac5ca41af917e9e02fb356ce8 (127.27.161.2:43345)
I20260810 03:26:41.758701 6563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6a73ce006274facbf8e092fd626fcd5" candidate_uuid: "05f07b3f00144d75b6c6fc9a40632f17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7227f1049a6433489fae09b53501439"
I20260810 03:26:41.758718 6775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6a73ce006274facbf8e092fd626fcd5" candidate_uuid: "05f07b3f00144d75b6c6fc9a40632f17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b851ca7ac5ca41af917e9e02fb356ce8"
I20260810 03:26:41.758922 6563 raft_consensus.cc:3037] T e6a73ce006274facbf8e092fd626fcd5 P b7227f1049a6433489fae09b53501439 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:26:41.758925 6775 raft_consensus.cc:3037] T e6a73ce006274facbf8e092fd626fcd5 P b851ca7ac5ca41af917e9e02fb356ce8 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:26:41.761329 6563 raft_consensus.cc:2454] T e6a73ce006274facbf8e092fd626fcd5 P b7227f1049a6433489fae09b53501439 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05f07b3f00144d75b6c6fc9a40632f17 in term 1.
I20260810 03:26:41.761809 6883 leader_election.cc:302] T e6a73ce006274facbf8e092fd626fcd5 P 05f07b3f00144d75b6c6fc9a40632f17 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05f07b3f00144d75b6c6fc9a40632f17, b7227f1049a6433489fae09b53501439; no voters:
I20260810 03:26:41.762152 7204 raft_consensus.cc:2781] T e6a73ce006274facbf8e092fd626fcd5 P 05f07b3f00144d75b6c6fc9a40632f17 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:26:41.762146 6775 raft_consensus.cc:2454] T e6a73ce006274facbf8e092fd626fcd5 P b851ca7ac5ca41af917e9e02fb356ce8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05f07b3f00144d75b6c6fc9a40632f17 in term 1.
I20260810 03:26:41.762921 7204 raft_consensus.cc:686] T e6a73ce006274facbf8e092fd626fcd5 P 05f07b3f00144d75b6c6fc9a40632f17 [term 1 LEADER]: Becoming Leader. State: Replica: 05f07b3f00144d75b6c6fc9a40632f17, State: Running, Role: LEADER
I20260810 03:26:41.763674 7204 consensus_queue.cc:237] T e6a73ce006274facbf8e092fd626fcd5 P 05f07b3f00144d75b6c6fc9a40632f17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f07b3f00144d75b6c6fc9a40632f17" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 45853 } } peers { permanent_uuid: "b7227f1049a6433489fae09b53501439" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 36623 } } peers { permanent_uuid: "b851ca7ac5ca41af917e9e02fb356ce8" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 43345 } }
I20260810 03:26:41.764120 7210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:41.763603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:36281 (local address 127.27.161.3:56075)
0810 03:26:41.763812 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0810 03:26:41.763941 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.62:36281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260810 03:26:41.767843 6346 catalog_manager.cc:5495] T e6a73ce006274facbf8e092fd626fcd5 P 05f07b3f00144d75b6c6fc9a40632f17 reported cstate change: term changed from 0 to 1, leader changed from <none> to 05f07b3f00144d75b6c6fc9a40632f17 (127.27.161.3). New cstate: current_term: 1 leader_uuid: "05f07b3f00144d75b6c6fc9a40632f17" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f07b3f00144d75b6c6fc9a40632f17" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 45853 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b7227f1049a6433489fae09b53501439" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 36623 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b851ca7ac5ca41af917e9e02fb356ce8" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 43345 } health_report { overall_health: UNKNOWN } } }
I20260810 03:26:41.832185 7185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:41.826231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57860 (local address 127.27.161.3:45853)
0810 03:26:41.826500 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0810 03:26:41.826512 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 03:26:41.826602 (+ 90us) server_negotiation.cc:407] Connection header received
0810 03:26:41.826852 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:26:41.826857 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:26:41.826949 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:26:41.827106 (+ 157us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 03:26:41.827519 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:41.828278 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:41.829835 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:41.830325 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:41.830454 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:41.830918 (+ 464us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 03:26:41.831521 (+ 603us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 03:26:41.831680 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0810 03:26:41.831791 (+ 111us) server_negotiation.cc:299] Negotiation successful
0810 03:26:41.831886 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260810 03:26:41.837344 6563 raft_consensus.cc:1261] T e6a73ce006274facbf8e092fd626fcd5 P b7227f1049a6433489fae09b53501439 [term 1 FOLLOWER]: Refusing update from remote peer 05f07b3f00144d75b6c6fc9a40632f17: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:26:41.837703 6775 raft_consensus.cc:1261] T e6a73ce006274facbf8e092fd626fcd5 P b851ca7ac5ca41af917e9e02fb356ce8 [term 1 FOLLOWER]: Refusing update from remote peer 05f07b3f00144d75b6c6fc9a40632f17: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:26:41.838204 7213 consensus_queue.cc:1035] T e6a73ce006274facbf8e092fd626fcd5 P 05f07b3f00144d75b6c6fc9a40632f17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7227f1049a6433489fae09b53501439" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 36623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:26:41.838544 7204 consensus_queue.cc:1035] T e6a73ce006274facbf8e092fd626fcd5 P 05f07b3f00144d75b6c6fc9a40632f17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b851ca7ac5ca41af917e9e02fb356ce8" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 43345 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:26:41.841434 7203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:41.840528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:36281 (local address 127.27.161.1:51627)
0810 03:26:41.841102 (+ 574us) negotiation.cc:107] Waiting for socket to connect
0810 03:26:41.841227 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.62:36281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":332}
I20260810 03:26:41.842602 7212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:41.842197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:36281 (local address 127.27.161.2:33877)
0810 03:26:41.842397 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0810 03:26:41.842473 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.62:36281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260810 03:26:41.851055 7210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:41.850099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:36281 (local address 127.27.161.3:35695)
0810 03:26:41.850751 (+ 652us) negotiation.cc:107] Waiting for socket to connect
0810 03:26:41.850848 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.62:36281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":531}
I20260810 03:26:41.851377 7203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:41.850909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:36281 (local address 127.27.161.1:38089)
0810 03:26:41.851128 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0810 03:26:41.851218 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.62:36281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117}
I20260810 03:26:41.853186 7212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:41.852682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:36281 (local address 127.27.161.2:41135)
0810 03:26:41.852929 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0810 03:26:41.853024 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.62:36281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133}
I20260810 03:26:41.866030 6345 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32976:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 03:26:41.866351 6345 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32976:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 03:26:41.873011 6345 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1245ea4d6f734899a3087452b0495ee2: Sending DeleteTablet for 3 replicas of tablet e6a73ce006274facbf8e092fd626fcd5
I20260810 03:26:41.874221 6953 tablet_service.cc:1514] Processing DeleteTablet for tablet e6a73ce006274facbf8e092fd626fcd5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:26:41 UTC) from {username='oryx', principal='oryx/127.27.161.60@KRBTEST.COM'} at 127.0.0.1:57858
I20260810 03:26:41.874306 6537 tablet_service.cc:1514] Processing DeleteTablet for tablet e6a73ce006274facbf8e092fd626fcd5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:26:41 UTC) from {username='oryx', principal='oryx/127.27.161.60@KRBTEST.COM'} at 127.0.0.1:55062
I20260810 03:26:41.874657 6742 tablet_service.cc:1514] Processing DeleteTablet for tablet e6a73ce006274facbf8e092fd626fcd5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:26:41 UTC) from {username='oryx', principal='oryx/127.27.161.60@KRBTEST.COM'} at 127.0.0.1:60430
I20260810 03:26:41.875073 7237 tablet_replica.cc:330] T e6a73ce006274facbf8e092fd626fcd5 P b7227f1049a6433489fae09b53501439: stopping tablet replica
I20260810 03:26:41.875334 7237 raft_consensus.cc:2229] T e6a73ce006274facbf8e092fd626fcd5 P b7227f1049a6433489fae09b53501439 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 03:26:41.876091 7237 raft_consensus.cc:2258] T e6a73ce006274facbf8e092fd626fcd5 P b7227f1049a6433489fae09b53501439 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:26:41.877178 28292 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iBH5/build/asan/bin/kudu pid 6332
I20260810 03:26:41.877817 7237 ts_tablet_manager.cc:1905] T e6a73ce006274facbf8e092fd626fcd5 P b7227f1049a6433489fae09b53501439: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:26:41.885174 7237 ts_tablet_manager.cc:1918] T e6a73ce006274facbf8e092fd626fcd5 P b7227f1049a6433489fae09b53501439: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 03:26:41.885458 7237 log.cc:1196] T e6a73ce006274facbf8e092fd626fcd5 P b7227f1049a6433489fae09b53501439: Deleting WAL directory at /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786332336319544-28292-0/minicluster-data/ts-0/wal/wals/e6a73ce006274facbf8e092fd626fcd5
I20260810 03:26:41.886219 7237 ts_tablet_manager.cc:1939] T e6a73ce006274facbf8e092fd626fcd5 P b7227f1049a6433489fae09b53501439: Deleting consensus metadata
I20260810 03:26:41.888926 6320 catalog_manager.cc:4841] TS b7227f1049a6433489fae09b53501439 (127.27.161.1:36623): tablet e6a73ce006274facbf8e092fd626fcd5 (table test-table [id=973af5d4d4ed49948553c21c1e4791ec]) successfully deleted
I20260810 03:26:41.894100 7236 tablet_replica.cc:330] T e6a73ce006274facbf8e092fd626fcd5 P 05f07b3f00144d75b6c6fc9a40632f17: stopping tablet replica
I20260810 03:26:41.894394 7236 raft_consensus.cc:2229] T e6a73ce006274facbf8e092fd626fcd5 P 05f07b3f00144d75b6c6fc9a40632f17 [term 1 LEADER]: Raft consensus shutting down.
I20260810 03:26:41.894936 7238 tablet_replica.cc:330] T e6a73ce006274facbf8e092fd626fcd5 P b851ca7ac5ca41af917e9e02fb356ce8: stopping tablet replica
I20260810 03:26:41.895140 7238 raft_consensus.cc:2229] T e6a73ce006274facbf8e092fd626fcd5 P b851ca7ac5ca41af917e9e02fb356ce8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 03:26:41.895450 7236 raft_consensus.cc:2258] T e6a73ce006274facbf8e092fd626fcd5 P 05f07b3f00144d75b6c6fc9a40632f17 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:26:41.895596 7238 raft_consensus.cc:2258] T e6a73ce006274facbf8e092fd626fcd5 P b851ca7ac5ca41af917e9e02fb356ce8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:26:41.896840 7238 ts_tablet_manager.cc:1905] T e6a73ce006274facbf8e092fd626fcd5 P b851ca7ac5ca41af917e9e02fb356ce8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:26:41.896816 7236 ts_tablet_manager.cc:1905] T e6a73ce006274facbf8e092fd626fcd5 P 05f07b3f00144d75b6c6fc9a40632f17: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:26:41.904280 7192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:41.878857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55064 (local address 127.27.161.1:36623)
0810 03:26:41.879050 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0810 03:26:41.879056 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:26:41.882823 (+ 3767us) server_negotiation.cc:407] Connection header received
0810 03:26:41.883509 (+ 686us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:26:41.883516 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:26:41.883620 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:26:41.883712 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:26:41.887384 (+ 3672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:41.888577 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:41.890475 (+ 1898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:41.891125 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:41.891258 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:41.895450 (+ 4192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:26:41.895472 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:26:41.895506 (+ 34us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:26:41.895561 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:26:41.899989 (+ 4428us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:26:41.901066 (+ 1077us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:26:41.901077 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:26:41.901081 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:26:41.901468 (+ 387us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:26:41.902025 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:26:41.902030 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:26:41.902034 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:26:41.902559 (+ 525us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:26:41.902856 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0810 03:26:41.903540 (+ 684us) server_negotiation.cc:299] Negotiation successful
0810 03:26:41.903897 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260810 03:26:41.904520 7238 ts_tablet_manager.cc:1918] T e6a73ce006274facbf8e092fd626fcd5 P b851ca7ac5ca41af917e9e02fb356ce8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 03:26:41.904735 7238 log.cc:1196] T e6a73ce006274facbf8e092fd626fcd5 P b851ca7ac5ca41af917e9e02fb356ce8: Deleting WAL directory at /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786332336319544-28292-0/minicluster-data/ts-1/wal/wals/e6a73ce006274facbf8e092fd626fcd5
I20260810 03:26:41.904790 7236 ts_tablet_manager.cc:1918] T e6a73ce006274facbf8e092fd626fcd5 P 05f07b3f00144d75b6c6fc9a40632f17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 03:26:41.905011 7236 log.cc:1196] T e6a73ce006274facbf8e092fd626fcd5 P 05f07b3f00144d75b6c6fc9a40632f17: Deleting WAL directory at /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786332336319544-28292-0/minicluster-data/ts-2/wal/wals/e6a73ce006274facbf8e092fd626fcd5
I20260810 03:26:41.905434 7238 ts_tablet_manager.cc:1939] T e6a73ce006274facbf8e092fd626fcd5 P b851ca7ac5ca41af917e9e02fb356ce8: Deleting consensus metadata
I20260810 03:26:41.905627 7236 ts_tablet_manager.cc:1939] T e6a73ce006274facbf8e092fd626fcd5 P 05f07b3f00144d75b6c6fc9a40632f17: Deleting consensus metadata
I20260810 03:26:41.906068 6623 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:26:41.907132 6322 catalog_manager.cc:4841] TS 05f07b3f00144d75b6c6fc9a40632f17 (127.27.161.3:45853): tablet e6a73ce006274facbf8e092fd626fcd5 (table test-table [id=973af5d4d4ed49948553c21c1e4791ec]) successfully deleted
I20260810 03:26:41.911681 6322 catalog_manager.cc:4841] TS b851ca7ac5ca41af917e9e02fb356ce8 (127.27.161.2:43345): tablet e6a73ce006274facbf8e092fd626fcd5 (table test-table [id=973af5d4d4ed49948553c21c1e4791ec]) successfully deleted
I20260810 03:26:41.914127 7162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:41.913629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:36281 (local address 127.0.0.1:37574)
0810 03:26:41.913831 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0810 03:26:41.913932 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.62:36281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260810 03:26:41.915274 7162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:41.914908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:36281 (local address 127.0.0.1:37580)
0810 03:26:41.915066 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0810 03:26:41.915143 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.62:36281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260810 03:26:42.086824 28292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iBH5/build/asan/bin/kudu with pid 6332
I20260810 03:26:42.141858 28292 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iBH5/build/asan/bin/kudu pid 6646
I20260810 03:26:42.161432 7194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:42.142991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60434 (local address 127.27.161.2:43345)
0810 03:26:42.143196 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0810 03:26:42.143202 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:26:42.143305 (+ 103us) server_negotiation.cc:407] Connection header received
0810 03:26:42.143499 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:26:42.143506 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:26:42.143600 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:26:42.143724 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:26:42.145572 (+ 1848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:42.146887 (+ 1315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:42.148829 (+ 1942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:42.149543 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:42.149682 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:42.153128 (+ 3446us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:26:42.153152 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:26:42.153177 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:26:42.153239 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:26:42.157793 (+ 4554us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:26:42.158770 (+ 977us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:26:42.158782 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:26:42.158785 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:26:42.158990 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:26:42.159403 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:26:42.159409 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:26:42.159412 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:26:42.160033 (+ 621us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:26:42.160370 (+ 337us) server_negotiation.cc:1035] Waiting for connection context
0810 03:26:42.160689 (+ 319us) server_negotiation.cc:299] Negotiation successful
0810 03:26:42.161022 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260810 03:26:42.162547 6835 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:26:42.333698 28292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iBH5/build/asan/bin/kudu with pid 6646
I20260810 03:26:42.377310 28292 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iBH5/build/asan/bin/kudu pid 6861
I20260810 03:26:42.399907 7243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:42.378264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57862 (local address 127.27.161.3:45853)
0810 03:26:42.378837 (+ 573us) server_negotiation.cc:206] Beginning negotiation
0810 03:26:42.378841 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:26:42.378859 (+ 18us) server_negotiation.cc:407] Connection header received
0810 03:26:42.378920 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:26:42.378923 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:26:42.378985 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:26:42.379077 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:26:42.381261 (+ 2184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:42.382660 (+ 1399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:42.384957 (+ 2297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:42.385735 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:42.385896 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:42.391052 (+ 5156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:26:42.391079 (+ 27us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:26:42.391104 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:26:42.391184 (+ 80us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:26:42.395522 (+ 4338us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:26:42.397032 (+ 1510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:26:42.397043 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:26:42.397046 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:26:42.397227 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:26:42.397751 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:26:42.397755 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:26:42.397758 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:26:42.398523 (+ 765us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:26:42.398895 (+ 372us) server_negotiation.cc:1035] Waiting for connection context
0810 03:26:42.399357 (+ 462us) server_negotiation.cc:299] Negotiation successful
0810 03:26:42.399579 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":455,"thread_start_us":275,"threads_started":1}
I20260810 03:26:42.400756 7046 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:26:42.427006 7245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:42.426166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:36281 (local address 127.0.0.1:37594)
0810 03:26:42.426726 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0810 03:26:42.426823 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.62:36281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":295,"threads_started":1}
I20260810 03:26:42.428592 7245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:42.428155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:36281 (local address 127.0.0.1:37596)
0810 03:26:42.428321 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0810 03:26:42.428404 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.62:36281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260810 03:26:42.546753 28292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iBH5/build/asan/bin/kudu with pid 6861
I20260810 03:26:42.574664 28292 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iBH5/build/asan/bin/kudu pid 5853
I20260810 03:26:42.575884 6117 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:26:42.710923 28292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iBH5/build/asan/bin/kudu with pid 5853
I20260810 03:26:42.742388 28292 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iBH5/build/asan/bin/kudu pid 6015
I20260810 03:26:42.743623 6393 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:26:42.874178 28292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iBH5/build/asan/bin/kudu with pid 6015
2026-08-10T03:26:42Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (11774 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786332336319544-28292-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 03:26:42 dist-test-slave-vpvm krb5kdc[7252](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 03:26:42 dist-test-slave-vpvm krb5kdc[7252](info): set up 2 sockets
Aug 10 03:26:42 dist-test-slave-vpvm krb5kdc[7252](info): commencing operation
krb5kdc: starting...
W20260810 03:26:44.162725 28292 mini_kdc.cc:121] Time spent starting KDC: real 1.231s user 0.004s sys 0.037s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786332336319544-28292-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786332336319544-28292-0/krb5kdc/test-user.keytab.
Aug 10 03:26:44 dist-test-slave-vpvm krb5kdc[7252](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332404, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T03:26:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T03:26:44Z Disabled control of system clock
WARNING: no policy specified for oryx/127.27.161.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.161.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.161.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786332336319544-28292-0/krb5kdc/oryx_127.27.161.62.keytab.
Entry for principal oryx/127.27.161.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786332336319544-28292-0/krb5kdc/oryx_127.27.161.62.keytab.
I20260810 03:26:44.469224 28292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iBH5/build/asan/bin/kudu
/tmp/dist-test-task24iBH5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786332336319544-28292-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786332336319544-28292-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786332336319544-28292-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786332336319544-28292-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.161.62:35965
--webserver_interface=127.27.161.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786332336319544-28292-0/krb5kdc/oryx_127.27.161.62.keytab
--principal=oryx/127.27.161.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.161.20:42595
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.161.62:35965
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786332336319544-28292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786332336319544-28292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786332336319544-28292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:26:44.846189 7270 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:26:44.846511 7270 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:26:44.846654 7270 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:26:44.854583 7270 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:26:44.854691 7270 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260810 03:26:44.854748 7270 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:26:44.854833 7270 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:26:44.854893 7270 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:26:44.855050 7270 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:26:44.866691 7270 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.161.20:42595
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786332336319544-28292-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786332336319544-28292-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.161.62:35965
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786332336319544-28292-0/krb5kdc/oryx_127.27.161.62.keytab
--principal=oryx/127.27.161.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.161.62:35965
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786332336319544-28292-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.161.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786332336319544-28292-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:26:44.868320 7270 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:26:44.869990 7270 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:26:44.883625 7276 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:26:44.883876 7275 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:26:44.883651 7278 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:26:44.884254 7270 server_base.cc:1034] running on GCE node
I20260810 03:26:44.885258 7270 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:26:44.886904 7270 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:26:44.888136 7270 hybrid_clock.cc:648] HybridClock initialized: now 1786332404888057 us; error 61 us; skew 500 ppm
I20260810 03:26:44.888486 7270 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:26:44 dist-test-slave-vpvm krb5kdc[7252](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332404, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.161.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:26:44.892234 7270 init.cc:377] Logged in from keytab as oryx/127.27.161.62@KRBTEST.COM (short username oryx)
I20260810 03:26:44.893896 7270 webserver.cc:460] Webserver started at http://127.27.161.62:36503/ using document root <none> and password file <none>
I20260810 03:26:44.894744 7270 fs_manager.cc:359] Metadata directory not provided
I20260810 03:26:44.894855 7270 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:26:44.895175 7270 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:26:44.898343 7270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786332336319544-28292-0/minicluster-data/master-0/data/instance:
uuid: "6882776d888e4399b85872d4661cecf3"
format_stamp: "Formatted at 2026-08-10 03:26:44 on dist-test-slave-vpvm"
server_key: "9ecf281505af68e352337b4589f2b004"
server_key_iv: "e5aa12df7ab7c45b32bbbdcb1b52f307"
server_key_version: "encryptionkey@0"
I20260810 03:26:44.899143 7270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786332336319544-28292-0/minicluster-data/master-0/wal/instance:
uuid: "6882776d888e4399b85872d4661cecf3"
format_stamp: "Formatted at 2026-08-10 03:26:44 on dist-test-slave-vpvm"
server_key: "9ecf281505af68e352337b4589f2b004"
server_key_iv: "e5aa12df7ab7c45b32bbbdcb1b52f307"
server_key_version: "encryptionkey@0"
I20260810 03:26:44.904500 7270 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260810 03:26:44.908393 7291 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:26:44.910073 7270 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 03:26:44.910243 7270 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786332336319544-28292-0/minicluster-data/master-0/data,/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786332336319544-28292-0/minicluster-data/master-0/wal
uuid: "6882776d888e4399b85872d4661cecf3"
format_stamp: "Formatted at 2026-08-10 03:26:44 on dist-test-slave-vpvm"
server_key: "9ecf281505af68e352337b4589f2b004"
server_key_iv: "e5aa12df7ab7c45b32bbbdcb1b52f307"
server_key_version: "encryptionkey@0"
I20260810 03:26:44.910445 7270 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786332336319544-28292-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786332336319544-28292-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786332336319544-28292-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:26:44.935860 7270 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:26:44.940531 7270 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:26:44.940828 7270 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:26:44.963948 7270 rpc_server.cc:307] RPC server started. Bound to: 127.27.161.62:35965
I20260810 03:26:44.963968 7396 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.161.62:35965 every 8 connection(s)
I20260810 03:26:44.965458 7270 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786332336319544-28292-0/minicluster-data/master-0/data/info.pb
I20260810 03:26:44.970024 7398 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:26:44.970794 28292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iBH5/build/asan/bin/kudu as pid 7270
I20260810 03:26:44.971019 28292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786332336319544-28292-0/minicluster-data/master-0/wal/instance
I20260810 03:26:44.971755 28292 external_mini_cluster.cc:1428] Setting key b4e5023f2f8542c97819516fa3d89a2e
I20260810 03:26:44.986150 7398 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6882776d888e4399b85872d4661cecf3: Bootstrap starting.
Aug 10 03:26:44 dist-test-slave-vpvm krb5kdc[7252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332404, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.27.161.62@KRBTEST.COM
I20260810 03:26:44.990365 7398 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6882776d888e4399b85872d4661cecf3: Neither blocks nor log segments found. Creating new log.
I20260810 03:26:44.991760 7398 log.cc:824] T 00000000000000000000000000000000 P 6882776d888e4399b85872d4661cecf3: Log is configured to *not* fsync() on all Append() calls
I20260810 03:26:44.996089 7398 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6882776d888e4399b85872d4661cecf3: No bootstrap required, opened a new log
I20260810 03:26:44.999536 7403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:44.973861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33354 (local address 127.27.161.62:35965)
0810 03:26:44.974832 (+ 971us) server_negotiation.cc:206] Beginning negotiation
0810 03:26:44.974846 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 03:26:44.974893 (+ 47us) server_negotiation.cc:407] Connection header received
0810 03:26:44.976650 (+ 1757us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:26:44.976681 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:26:44.977150 (+ 469us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:26:44.977533 (+ 383us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:26:44.979537 (+ 2004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:44.981149 (+ 1612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:44.983510 (+ 2361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:44.984410 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:44.984574 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:44.989775 (+ 5201us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:26:44.989785 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:26:44.989802 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:26:44.989849 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:26:44.993642 (+ 3793us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:26:44.994694 (+ 1052us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:26:44.994702 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:26:44.994709 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:26:44.995010 (+ 301us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:26:44.995675 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:26:44.995679 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:26:44.995681 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:26:44.996368 (+ 687us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:26:44.996718 (+ 350us) server_negotiation.cc:1035] Waiting for connection context
0810 03:26:44.997419 (+ 701us) server_negotiation.cc:299] Negotiation successful
0810 03:26:44.998279 (+ 860us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":741,"thread_start_us":443,"threads_started":1}
I20260810 03:26:45.005679 7398 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6882776d888e4399b85872d4661cecf3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6882776d888e4399b85872d4661cecf3" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 35965 } }
I20260810 03:26:45.006106 7398 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6882776d888e4399b85872d4661cecf3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:26:45.006217 7398 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6882776d888e4399b85872d4661cecf3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6882776d888e4399b85872d4661cecf3, State: Initialized, Role: FOLLOWER
I20260810 03:26:45.006894 7398 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6882776d888e4399b85872d4661cecf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6882776d888e4399b85872d4661cecf3" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 35965 } }
I20260810 03:26:45.007139 7398 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6882776d888e4399b85872d4661cecf3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 03:26:45.007277 7398 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6882776d888e4399b85872d4661cecf3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 03:26:45.007468 7398 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6882776d888e4399b85872d4661cecf3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:26:45.009881 7398 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6882776d888e4399b85872d4661cecf3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6882776d888e4399b85872d4661cecf3" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 35965 } }
I20260810 03:26:45.010480 7398 leader_election.cc:302] T 00000000000000000000000000000000 P 6882776d888e4399b85872d4661cecf3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6882776d888e4399b85872d4661cecf3; no voters:
I20260810 03:26:45.011111 7398 leader_election.cc:288] T 00000000000000000000000000000000 P 6882776d888e4399b85872d4661cecf3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 03:26:45.011245 7405 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6882776d888e4399b85872d4661cecf3 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:26:45.011957 7405 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6882776d888e4399b85872d4661cecf3 [term 1 LEADER]: Becoming Leader. State: Replica: 6882776d888e4399b85872d4661cecf3, State: Running, Role: LEADER
I20260810 03:26:45.012509 7405 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6882776d888e4399b85872d4661cecf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6882776d888e4399b85872d4661cecf3" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 35965 } }
I20260810 03:26:45.013221 7398 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6882776d888e4399b85872d4661cecf3 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:26:45.015740 7407 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6882776d888e4399b85872d4661cecf3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6882776d888e4399b85872d4661cecf3. Latest consensus state: current_term: 1 leader_uuid: "6882776d888e4399b85872d4661cecf3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6882776d888e4399b85872d4661cecf3" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 35965 } } }
I20260810 03:26:45.016235 7407 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6882776d888e4399b85872d4661cecf3 [sys.catalog]: This master's current role is: LEADER
I20260810 03:26:45.017321 7413 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 03:26:45.017176 7406 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6882776d888e4399b85872d4661cecf3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6882776d888e4399b85872d4661cecf3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6882776d888e4399b85872d4661cecf3" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 35965 } } }
I20260810 03:26:45.017618 7406 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6882776d888e4399b85872d4661cecf3 [sys.catalog]: This master's current role is: LEADER
I20260810 03:26:45.026559 7413 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 03:26:45.037632 7413 catalog_manager.cc:1347] Generated new cluster ID: ea80146129b54090bf40b2f06a31fb51
I20260810 03:26:45.037767 7413 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 03:26:45.061064 7413 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 03:26:45.062569 7413 catalog_manager.cc:1504] Loading token signing keys...
I20260810 03:26:45.072503 7413 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6882776d888e4399b85872d4661cecf3: Generated new TSK 0
I20260810 03:26:45.073779 7413 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.27.161.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.161.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.161.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786332336319544-28292-0/krb5kdc/oryx_127.27.161.1.keytab.
Entry for principal oryx/127.27.161.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786332336319544-28292-0/krb5kdc/oryx_127.27.161.1.keytab.
I20260810 03:26:45.151574 28292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iBH5/build/asan/bin/kudu
/tmp/dist-test-task24iBH5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786332336319544-28292-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786332336319544-28292-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786332336319544-28292-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786332336319544-28292-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.161.1:0
--local_ip_for_outbound_sockets=127.27.161.1
--webserver_interface=127.27.161.1
--webserver_port=0
--tserver_master_addrs=127.27.161.62:35965
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786332336319544-28292-0/krb5kdc/oryx_127.27.161.1.keytab
--principal=oryx/127.27.161.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.161.20:42595
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786332336319544-28292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786332336319544-28292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786332336319544-28292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:26:45.534416 7436 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:26:45.534735 7436 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:26:45.534966 7436 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:26:45.542901 7436 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:26:45.543027 7436 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:26:45.543149 7436 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 03:26:45.543316 7436 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.161.1
I20260810 03:26:45.555015 7436 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.161.20:42595
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786332336319544-28292-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786332336319544-28292-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786332336319544-28292-0/krb5kdc/oryx_127.27.161.1.keytab
--principal=oryx/127.27.161.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.161.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786332336319544-28292-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.161.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.27.161.62:35965
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.161.1
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786332336319544-28292-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:26:45.556643 7436 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:26:45.558059 7436 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:26:45.572249 7443 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:26:45.572393 7442 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:26:45.572826 7446 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:26:45.573079 7436 server_base.cc:1034] running on GCE node
I20260810 03:26:45.573750 7436 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:26:45.574834 7436 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:26:45.576041 7436 hybrid_clock.cc:648] HybridClock initialized: now 1786332405575980 us; error 39 us; skew 500 ppm
I20260810 03:26:45.576337 7436 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:26:45 dist-test-slave-vpvm krb5kdc[7252](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332405, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.161.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:26:45.579916 7436 init.cc:377] Logged in from keytab as oryx/127.27.161.1@KRBTEST.COM (short username oryx)
I20260810 03:26:45.581674 7436 webserver.cc:460] Webserver started at http://127.27.161.1:42995/ using document root <none> and password file <none>
I20260810 03:26:45.582521 7436 fs_manager.cc:359] Metadata directory not provided
I20260810 03:26:45.582614 7436 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:26:45.582952 7436 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:26:45.586380 7436 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786332336319544-28292-0/minicluster-data/ts-0/data/instance:
uuid: "51a5b515913a4b10bd40695507e0fcca"
format_stamp: "Formatted at 2026-08-10 03:26:45 on dist-test-slave-vpvm"
server_key: "5fcc61341e33829c6e92cabdedf12a4f"
server_key_iv: "cab3230bce5d7aa042607db7af866e3c"
server_key_version: "encryptionkey@0"
I20260810 03:26:45.587368 7436 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786332336319544-28292-0/minicluster-data/ts-0/wal/instance:
uuid: "51a5b515913a4b10bd40695507e0fcca"
format_stamp: "Formatted at 2026-08-10 03:26:45 on dist-test-slave-vpvm"
server_key: "5fcc61341e33829c6e92cabdedf12a4f"
server_key_iv: "cab3230bce5d7aa042607db7af866e3c"
server_key_version: "encryptionkey@0"
I20260810 03:26:45.593146 7436 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260810 03:26:45.597103 7459 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:26:45.598748 7436 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 03:26:45.598913 7436 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786332336319544-28292-0/minicluster-data/ts-0/data,/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786332336319544-28292-0/minicluster-data/ts-0/wal
uuid: "51a5b515913a4b10bd40695507e0fcca"
format_stamp: "Formatted at 2026-08-10 03:26:45 on dist-test-slave-vpvm"
server_key: "5fcc61341e33829c6e92cabdedf12a4f"
server_key_iv: "cab3230bce5d7aa042607db7af866e3c"
server_key_version: "encryptionkey@0"
I20260810 03:26:45.599149 7436 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786332336319544-28292-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786332336319544-28292-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786332336319544-28292-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:26:45.617882 7436 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:26:45.622901 7436 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:26:45.623168 7436 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:26:45.626351 7436 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:26:45.626459 7436 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:26:45.626642 7436 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:26:45.626713 7436 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 03:26:45 dist-test-slave-vpvm krb5kdc[7252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332405, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.161.1@KRBTEST.COM for oryx/127.27.161.62@KRBTEST.COM
I20260810 03:26:45.654685 7476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:45.627814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:35965 (local address 127.27.161.1:45627)
0810 03:26:45.628873 (+ 1059us) negotiation.cc:107] Waiting for socket to connect
0810 03:26:45.628918 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0810 03:26:45.631269 (+ 2351us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:26:45.632523 (+ 1254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:26:45.632535 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:26:45.633567 (+ 1032us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:26:45.634594 (+ 1027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:26:45.634624 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:26:45.636182 (+ 1558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:26:45.636195 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:26:45.638545 (+ 2350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:26:45.638558 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:26:45.639833 (+ 1275us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:26:45.639843 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:26:45.640155 (+ 312us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:45.640991 (+ 836us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:26:45.641031 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:26:45.644772 (+ 3741us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:26:45.648698 (+ 3926us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:26:45.648710 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:26:45.648735 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:26:45.649463 (+ 728us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:26:45.650062 (+ 599us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:26:45.650066 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:26:45.650068 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:26:45.650328 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:26:45.651036 (+ 708us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:26:45.651049 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:26:45.651623 (+ 574us) client_negotiation.cc:769] Sending connection context
0810 03:26:45.652198 (+ 575us) client_negotiation.cc:240] Negotiation successful
0810 03:26:45.652809 (+ 611us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":722,"thread_start_us":416,"threads_started":1}
I20260810 03:26:45.656139 7475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:45.627161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.161.1:45627 (local address 127.27.161.62:35965)
0810 03:26:45.627763 (+ 602us) server_negotiation.cc:206] Beginning negotiation
0810 03:26:45.627769 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:26:45.629111 (+ 1342us) server_negotiation.cc:407] Connection header received
0810 03:26:45.631786 (+ 2675us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:26:45.631793 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:26:45.632062 (+ 269us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:26:45.632222 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:26:45.634853 (+ 2631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:45.635980 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:45.638688 (+ 2708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:45.639414 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:45.639613 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:45.645096 (+ 5483us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:26:45.645102 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:26:45.645105 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:26:45.645150 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:26:45.648500 (+ 3350us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:26:45.649667 (+ 1167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:26:45.649675 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:26:45.649679 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:26:45.649905 (+ 226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:26:45.650450 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:26:45.650455 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:26:45.650459 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:26:45.650886 (+ 427us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:26:45.651087 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0810 03:26:45.655644 (+ 4557us) server_negotiation.cc:299] Negotiation successful
0810 03:26:45.655850 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":470,"thread_start_us":278,"threads_started":1}
I20260810 03:26:45.673106 7436 rpc_server.cc:307] RPC server started. Bound to: 127.27.161.1:35325
I20260810 03:26:45.673122 7643 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.161.1:35325 every 8 connection(s)
I20260810 03:26:45.674645 7436 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786332336319544-28292-0/minicluster-data/ts-0/data/info.pb
I20260810 03:26:45.677294 7644 heartbeater.cc:347] Connected to a master server at 127.27.161.62:35965
I20260810 03:26:45.677573 7644 heartbeater.cc:464] Registering TS with master...
I20260810 03:26:45.678632 7644 heartbeater.cc:511] Master 127.27.161.62:35965 requested a full tablet report, sending...
I20260810 03:26:45.678964 7475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:45.663795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.161.1:34163 (local address 127.27.161.62:35965)
0810 03:26:45.663983 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0810 03:26:45.663989 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:26:45.664749 (+ 760us) server_negotiation.cc:407] Connection header received
0810 03:26:45.665037 (+ 288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:26:45.665043 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:26:45.665125 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:26:45.665183 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:26:45.666639 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:45.667663 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:45.669713 (+ 2050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:45.670556 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:45.670678 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:45.673036 (+ 2358us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:26:45.673041 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:26:45.673045 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:26:45.673080 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:26:45.676000 (+ 2920us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:26:45.676798 (+ 798us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:26:45.676803 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:26:45.676806 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:26:45.676975 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:26:45.677323 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:26:45.677327 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:26:45.677329 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:26:45.677697 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:26:45.677817 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0810 03:26:45.678359 (+ 542us) server_negotiation.cc:299] Negotiation successful
0810 03:26:45.678660 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260810 03:26:45.679124 7606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:45.663858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:35965 (local address 127.27.161.1:34163)
0810 03:26:45.664625 (+ 767us) negotiation.cc:107] Waiting for socket to connect
0810 03:26:45.664652 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0810 03:26:45.664885 (+ 233us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:26:45.665351 (+ 466us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:26:45.665354 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:26:45.665990 (+ 636us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:26:45.666478 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:26:45.666491 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:26:45.667825 (+ 1334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:26:45.667830 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:26:45.669540 (+ 1710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:26:45.669558 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:26:45.670969 (+ 1411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:26:45.670976 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:26:45.671100 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:45.671772 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:26:45.671798 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:26:45.672822 (+ 1024us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:26:45.676145 (+ 3323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:26:45.676151 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:26:45.676155 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:26:45.676669 (+ 514us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:26:45.677084 (+ 415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:26:45.677088 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:26:45.677091 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:26:45.677217 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:26:45.677809 (+ 592us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:26:45.677815 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:26:45.678067 (+ 252us) client_negotiation.cc:769] Sending connection context
0810 03:26:45.678241 (+ 174us) client_negotiation.cc:240] Negotiation successful
0810 03:26:45.678675 (+ 434us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":596,"thread_start_us":427,"threads_started":1}
I20260810 03:26:45.681470 7319 ts_manager.cc:194] Registered new tserver with Master: 51a5b515913a4b10bd40695507e0fcca (127.27.161.1:35325)
I20260810 03:26:45.683501 7319 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.27.161.1@KRBTEST.COM'} at 127.27.161.1:45627
I20260810 03:26:45.684080 28292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iBH5/build/asan/bin/kudu as pid 7436
I20260810 03:26:45.684310 28292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786332336319544-28292-0/minicluster-data/ts-0/wal/instance
I20260810 03:26:45.685149 28292 external_mini_cluster.cc:1428] Setting key 75e64b1e3419a8b644b8e097c7db0065
WARNING: no policy specified for oryx/127.27.161.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.161.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.161.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786332336319544-28292-0/krb5kdc/oryx_127.27.161.2.keytab.
Entry for principal oryx/127.27.161.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786332336319544-28292-0/krb5kdc/oryx_127.27.161.2.keytab.
I20260810 03:26:45.753595 28292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iBH5/build/asan/bin/kudu
/tmp/dist-test-task24iBH5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786332336319544-28292-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786332336319544-28292-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786332336319544-28292-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786332336319544-28292-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.161.2:0
--local_ip_for_outbound_sockets=127.27.161.2
--webserver_interface=127.27.161.2
--webserver_port=0
--tserver_master_addrs=127.27.161.62:35965
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786332336319544-28292-0/krb5kdc/oryx_127.27.161.2.keytab
--principal=oryx/127.27.161.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.161.20:42595
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786332336319544-28292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786332336319544-28292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786332336319544-28292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:26:46.121699 7650 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:26:46.122069 7650 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:26:46.122220 7650 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:26:46.129784 7650 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:26:46.129935 7650 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:26:46.130088 7650 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 03:26:46.130213 7650 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.161.2
I20260810 03:26:46.140417 7650 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.161.20:42595
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786332336319544-28292-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786332336319544-28292-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786332336319544-28292-0/krb5kdc/oryx_127.27.161.2.keytab
--principal=oryx/127.27.161.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.161.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786332336319544-28292-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.161.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.27.161.62:35965
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.161.2
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786332336319544-28292-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:26:46.141769 7650 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:26:46.143146 7650 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:26:46.156301 7664 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:26:46.156348 7661 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:26:46.156262 7660 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:26:46.157450 7650 server_base.cc:1034] running on GCE node
I20260810 03:26:46.158102 7650 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:26:46.159024 7650 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:26:46.160236 7650 hybrid_clock.cc:648] HybridClock initialized: now 1786332406160158 us; error 52 us; skew 500 ppm
I20260810 03:26:46.160516 7650 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:26:46 dist-test-slave-vpvm krb5kdc[7252](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332406, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.161.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:26:46.163627 7650 init.cc:377] Logged in from keytab as oryx/127.27.161.2@KRBTEST.COM (short username oryx)
I20260810 03:26:46.165084 7650 webserver.cc:460] Webserver started at http://127.27.161.2:43725/ using document root <none> and password file <none>
I20260810 03:26:46.165796 7650 fs_manager.cc:359] Metadata directory not provided
I20260810 03:26:46.165900 7650 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:26:46.166239 7650 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:26:46.169170 7650 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786332336319544-28292-0/minicluster-data/ts-1/data/instance:
uuid: "db6b5c27fe7b480f96dbf4e5afd7b6b9"
format_stamp: "Formatted at 2026-08-10 03:26:46 on dist-test-slave-vpvm"
server_key: "899c40689a13d9d1ba8549fa179fc38f"
server_key_iv: "ed0a6ab3f1d1440c80f7bd034a66d326"
server_key_version: "encryptionkey@0"
I20260810 03:26:46.170009 7650 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786332336319544-28292-0/minicluster-data/ts-1/wal/instance:
uuid: "db6b5c27fe7b480f96dbf4e5afd7b6b9"
format_stamp: "Formatted at 2026-08-10 03:26:46 on dist-test-slave-vpvm"
server_key: "899c40689a13d9d1ba8549fa179fc38f"
server_key_iv: "ed0a6ab3f1d1440c80f7bd034a66d326"
server_key_version: "encryptionkey@0"
I20260810 03:26:46.175335 7650 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.002s
I20260810 03:26:46.179036 7673 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:26:46.180545 7650 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260810 03:26:46.180730 7650 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786332336319544-28292-0/minicluster-data/ts-1/data,/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786332336319544-28292-0/minicluster-data/ts-1/wal
uuid: "db6b5c27fe7b480f96dbf4e5afd7b6b9"
format_stamp: "Formatted at 2026-08-10 03:26:46 on dist-test-slave-vpvm"
server_key: "899c40689a13d9d1ba8549fa179fc38f"
server_key_iv: "ed0a6ab3f1d1440c80f7bd034a66d326"
server_key_version: "encryptionkey@0"
I20260810 03:26:46.180949 7650 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786332336319544-28292-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786332336319544-28292-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786332336319544-28292-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:26:46.192780 7650 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:26:46.197084 7650 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:26:46.197340 7650 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:26:46.200119 7650 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:26:46.200209 7650 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:26:46.200382 7650 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:26:46.200448 7650 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 03:26:46 dist-test-slave-vpvm krb5kdc[7252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332406, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.161.2@KRBTEST.COM for oryx/127.27.161.62@KRBTEST.COM
I20260810 03:26:46.226107 7685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:46.200551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.161.2:37351 (local address 127.27.161.62:35965)
0810 03:26:46.201350 (+ 799us) server_negotiation.cc:206] Beginning negotiation
0810 03:26:46.201357 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:26:46.202358 (+ 1001us) server_negotiation.cc:407] Connection header received
0810 03:26:46.204789 (+ 2431us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:26:46.204795 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:26:46.204903 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:26:46.205000 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:26:46.207507 (+ 2507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:46.208981 (+ 1474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:46.211989 (+ 3008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:46.212480 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:46.212625 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:46.218845 (+ 6220us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:26:46.218853 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:26:46.218857 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:26:46.218907 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:26:46.222234 (+ 3327us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:26:46.223075 (+ 841us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:26:46.223081 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:26:46.223085 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:26:46.223267 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:26:46.223865 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:26:46.223872 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:26:46.223875 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:26:46.224326 (+ 451us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:26:46.224516 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0810 03:26:46.225329 (+ 813us) server_negotiation.cc:299] Negotiation successful
0810 03:26:46.225620 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":663,"thread_start_us":547,"threads_started":1}
I20260810 03:26:46.226891 7686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:46.201228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:35965 (local address 127.27.161.2:37351)
0810 03:26:46.202162 (+ 934us) negotiation.cc:107] Waiting for socket to connect
0810 03:26:46.202213 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0810 03:26:46.204491 (+ 2278us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:26:46.205333 (+ 842us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:26:46.205346 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:26:46.206336 (+ 990us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:26:46.207243 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:26:46.207276 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:26:46.209229 (+ 1953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:26:46.209235 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:26:46.211787 (+ 2552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:26:46.211805 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:26:46.212784 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:26:46.212799 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:26:46.213271 (+ 472us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:46.214381 (+ 1110us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:26:46.214430 (+ 49us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:26:46.218099 (+ 3669us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:26:46.222409 (+ 4310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:26:46.222420 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:26:46.222439 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:26:46.222926 (+ 487us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:26:46.223424 (+ 498us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:26:46.223430 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:26:46.223434 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:26:46.223709 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:26:46.224497 (+ 788us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:26:46.224506 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:26:46.224916 (+ 410us) client_negotiation.cc:769] Sending connection context
0810 03:26:46.225172 (+ 256us) client_negotiation.cc:240] Negotiation successful
0810 03:26:46.225610 (+ 438us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":609,"thread_start_us":392,"threads_started":1}
I20260810 03:26:46.248406 7775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:46.232569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:35965 (local address 127.27.161.2:57289)
0810 03:26:46.233232 (+ 663us) negotiation.cc:107] Waiting for socket to connect
0810 03:26:46.233250 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 03:26:46.233374 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:26:46.233734 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:26:46.233738 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:26:46.234358 (+ 620us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:26:46.234901 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:26:46.234915 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:26:46.236163 (+ 1248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:26:46.236166 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:26:46.237831 (+ 1665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:26:46.237848 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:26:46.238994 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:26:46.239003 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:26:46.239216 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:46.240103 (+ 887us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:26:46.240131 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:26:46.241461 (+ 1330us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:26:46.244758 (+ 3297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:26:46.244765 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:26:46.244770 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:26:46.245387 (+ 617us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:26:46.245836 (+ 449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:26:46.245840 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:26:46.245845 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:26:46.246056 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:26:46.246934 (+ 878us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:26:46.246945 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:26:46.247210 (+ 265us) client_negotiation.cc:769] Sending connection context
0810 03:26:46.247388 (+ 178us) client_negotiation.cc:240] Negotiation successful
0810 03:26:46.247807 (+ 419us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":511,"thread_start_us":345,"threads_started":1}
I20260810 03:26:46.249420 7685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:46.232608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.161.2:57289 (local address 127.27.161.62:35965)
0810 03:26:46.232815 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0810 03:26:46.232822 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:26:46.233326 (+ 504us) server_negotiation.cc:407] Connection header received
0810 03:26:46.233480 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:26:46.233485 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:26:46.233575 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:26:46.233689 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:26:46.235058 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:46.235992 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:46.238140 (+ 2148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:46.238698 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:46.238834 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:46.241683 (+ 2849us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:26:46.241688 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:26:46.241691 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:26:46.241728 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:26:46.244609 (+ 2881us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:26:46.245558 (+ 949us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:26:46.245563 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:26:46.245566 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:26:46.245717 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:26:46.246203 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:26:46.246209 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:26:46.246213 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:26:46.246658 (+ 445us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:26:46.246847 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0810 03:26:46.248950 (+ 2103us) server_negotiation.cc:299] Negotiation successful
0810 03:26:46.249160 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260810 03:26:46.259100 7650 rpc_server.cc:307] RPC server started. Bound to: 127.27.161.2:33609
I20260810 03:26:46.259120 7843 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.161.2:33609 every 8 connection(s)
I20260810 03:26:46.260401 7650 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786332336319544-28292-0/minicluster-data/ts-1/data/info.pb
I20260810 03:26:46.262548 7844 heartbeater.cc:347] Connected to a master server at 127.27.161.62:35965
I20260810 03:26:46.262794 7844 heartbeater.cc:464] Registering TS with master...
I20260810 03:26:46.263495 7844 heartbeater.cc:511] Master 127.27.161.62:35965 requested a full tablet report, sending...
I20260810 03:26:46.264739 28292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iBH5/build/asan/bin/kudu as pid 7650
I20260810 03:26:46.264909 28292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786332336319544-28292-0/minicluster-data/ts-1/wal/instance
I20260810 03:26:46.264981 7319 ts_manager.cc:194] Registered new tserver with Master: db6b5c27fe7b480f96dbf4e5afd7b6b9 (127.27.161.2:33609)
I20260810 03:26:46.265473 28292 external_mini_cluster.cc:1428] Setting key a3b66a42b039f3fb90af63d03db5e9a5
I20260810 03:26:46.266114 7319 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.27.161.2@KRBTEST.COM'} at 127.27.161.2:37351
WARNING: no policy specified for oryx/127.27.161.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.161.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.161.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786332336319544-28292-0/krb5kdc/oryx_127.27.161.3.keytab.
Entry for principal oryx/127.27.161.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786332336319544-28292-0/krb5kdc/oryx_127.27.161.3.keytab.
I20260810 03:26:46.325372 28292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iBH5/build/asan/bin/kudu
/tmp/dist-test-task24iBH5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786332336319544-28292-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786332336319544-28292-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786332336319544-28292-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786332336319544-28292-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.161.3:0
--local_ip_for_outbound_sockets=127.27.161.3
--webserver_interface=127.27.161.3
--webserver_port=0
--tserver_master_addrs=127.27.161.62:35965
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786332336319544-28292-0/krb5kdc/oryx_127.27.161.3.keytab
--principal=oryx/127.27.161.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.161.20:42595
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786332336319544-28292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786332336319544-28292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786332336319544-28292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:26:46.677865 7851 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:26:46.678208 7851 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:26:46.678355 7851 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:26:46.686082 7851 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:26:46.686206 7851 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:26:46.686300 7851 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 03:26:46.686386 7851 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.161.3
I20260810 03:26:46.687626 7644 heartbeater.cc:503] Master 127.27.161.62:35965 was elected leader, sending a full tablet report...
I20260810 03:26:46.697103 7851 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.161.20:42595
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786332336319544-28292-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786332336319544-28292-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786332336319544-28292-0/krb5kdc/oryx_127.27.161.3.keytab
--principal=oryx/127.27.161.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.161.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786332336319544-28292-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.161.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.27.161.62:35965
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.161.3
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786332336319544-28292-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:26:46.698585 7851 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:26:46.699986 7851 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:26:46.713749 7859 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:26:46.713721 7860 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:26:46.714085 7862 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:26:46.714834 7851 server_base.cc:1034] running on GCE node
I20260810 03:26:46.715488 7851 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:26:46.716462 7851 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:26:46.717705 7851 hybrid_clock.cc:648] HybridClock initialized: now 1786332406717629 us; error 51 us; skew 500 ppm
I20260810 03:26:46.718070 7851 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:26:46 dist-test-slave-vpvm krb5kdc[7252](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332406, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.161.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:26:46.726146 7851 init.cc:377] Logged in from keytab as oryx/127.27.161.3@KRBTEST.COM (short username oryx)
I20260810 03:26:46.727952 7851 webserver.cc:460] Webserver started at http://127.27.161.3:36299/ using document root <none> and password file <none>
I20260810 03:26:46.728825 7851 fs_manager.cc:359] Metadata directory not provided
I20260810 03:26:46.728974 7851 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:26:46.729341 7851 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:26:46.732681 7851 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786332336319544-28292-0/minicluster-data/ts-2/data/instance:
uuid: "9a1b431e2e804faab2004da408cd20f8"
format_stamp: "Formatted at 2026-08-10 03:26:46 on dist-test-slave-vpvm"
server_key: "7a8f160ff85eb3b7b265b4f1e250a7d5"
server_key_iv: "890f597c602417ebf249c2f8abb5d423"
server_key_version: "encryptionkey@0"
I20260810 03:26:46.733575 7851 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786332336319544-28292-0/minicluster-data/ts-2/wal/instance:
uuid: "9a1b431e2e804faab2004da408cd20f8"
format_stamp: "Formatted at 2026-08-10 03:26:46 on dist-test-slave-vpvm"
server_key: "7a8f160ff85eb3b7b265b4f1e250a7d5"
server_key_iv: "890f597c602417ebf249c2f8abb5d423"
server_key_version: "encryptionkey@0"
I20260810 03:26:46.740144 7851 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260810 03:26:46.743995 7874 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:26:46.745462 7851 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 03:26:46.745638 7851 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786332336319544-28292-0/minicluster-data/ts-2/data,/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786332336319544-28292-0/minicluster-data/ts-2/wal
uuid: "9a1b431e2e804faab2004da408cd20f8"
format_stamp: "Formatted at 2026-08-10 03:26:46 on dist-test-slave-vpvm"
server_key: "7a8f160ff85eb3b7b265b4f1e250a7d5"
server_key_iv: "890f597c602417ebf249c2f8abb5d423"
server_key_version: "encryptionkey@0"
I20260810 03:26:46.745919 7851 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786332336319544-28292-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786332336319544-28292-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786332336319544-28292-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:26:46.760989 7851 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:26:46.766741 7851 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:26:46.767033 7851 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:26:46.770187 7851 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:26:46.770287 7851 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:26:46.770437 7851 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:26:46.770501 7851 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 03:26:46 dist-test-slave-vpvm krb5kdc[7252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332406, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.161.3@KRBTEST.COM for oryx/127.27.161.62@KRBTEST.COM
I20260810 03:26:46.797468 7893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:46.771566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:35965 (local address 127.27.161.3:40287)
0810 03:26:46.772501 (+ 935us) negotiation.cc:107] Waiting for socket to connect
0810 03:26:46.772565 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0810 03:26:46.774824 (+ 2259us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:26:46.775779 (+ 955us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:26:46.775788 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:26:46.776745 (+ 957us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:26:46.777877 (+ 1132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:26:46.777913 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:26:46.779632 (+ 1719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:26:46.779637 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:26:46.781810 (+ 2173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:26:46.781827 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:26:46.783032 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:26:46.783040 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:26:46.783285 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:46.784027 (+ 742us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:26:46.784069 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:26:46.787253 (+ 3184us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:26:46.791744 (+ 4491us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:26:46.791759 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:26:46.791776 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:26:46.792559 (+ 783us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:26:46.793014 (+ 455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:26:46.793024 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:26:46.793028 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:26:46.793374 (+ 346us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:26:46.794096 (+ 722us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:26:46.794105 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:26:46.794726 (+ 621us) client_negotiation.cc:769] Sending connection context
0810 03:26:46.795132 (+ 406us) client_negotiation.cc:240] Negotiation successful
0810 03:26:46.795741 (+ 609us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":637,"thread_start_us":399,"threads_started":1}
I20260810 03:26:46.799160 7891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:46.770930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.161.3:40287 (local address 127.27.161.62:35965)
0810 03:26:46.771495 (+ 565us) server_negotiation.cc:206] Beginning negotiation
0810 03:26:46.771502 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:26:46.773013 (+ 1511us) server_negotiation.cc:407] Connection header received
0810 03:26:46.775171 (+ 2158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:26:46.775177 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:26:46.775282 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:26:46.775386 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:26:46.778195 (+ 2809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:46.779419 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:46.782020 (+ 2601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:46.782722 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:46.782878 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:46.787627 (+ 4749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:26:46.787636 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:26:46.787641 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:26:46.787694 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:26:46.791512 (+ 3818us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:26:46.792723 (+ 1211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:26:46.792730 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:26:46.792733 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:26:46.792885 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:26:46.793508 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:26:46.793512 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:26:46.793514 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:26:46.793923 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:26:46.794104 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0810 03:26:46.798474 (+ 4370us) server_negotiation.cc:299] Negotiation successful
0810 03:26:46.798750 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":434,"thread_start_us":302,"threads_started":1}
I20260810 03:26:46.820734 7851 rpc_server.cc:307] RPC server started. Bound to: 127.27.161.3:40179
I20260810 03:26:46.820881 8018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:46.805571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:35965 (local address 127.27.161.3:35765)
0810 03:26:46.806291 (+ 720us) negotiation.cc:107] Waiting for socket to connect
0810 03:26:46.806313 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0810 03:26:46.806473 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:26:46.806806 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:26:46.806811 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:26:46.807297 (+ 486us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:26:46.807616 (+ 319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:26:46.807627 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:26:46.808658 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:26:46.808662 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:26:46.810071 (+ 1409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:26:46.810090 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:26:46.811524 (+ 1434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:26:46.811531 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:26:46.811743 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:46.812541 (+ 798us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:26:46.812570 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:26:46.813831 (+ 1261us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:26:46.817336 (+ 3505us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:26:46.817343 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:26:46.817348 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:26:46.818032 (+ 684us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:26:46.818532 (+ 500us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:26:46.818538 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:26:46.818542 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:26:46.818756 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:26:46.819289 (+ 533us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:26:46.819295 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:26:46.819656 (+ 361us) client_negotiation.cc:769] Sending connection context
0810 03:26:46.819823 (+ 167us) client_negotiation.cc:240] Negotiation successful
0810 03:26:46.820275 (+ 452us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":585,"thread_start_us":431,"threads_started":1}
I20260810 03:26:46.820775 8050 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.161.3:40179 every 8 connection(s)
I20260810 03:26:46.821383 7891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:46.805475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.161.3:35765 (local address 127.27.161.62:35965)
0810 03:26:46.805637 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0810 03:26:46.805644 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:26:46.806419 (+ 775us) server_negotiation.cc:407] Connection header received
0810 03:26:46.806598 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:26:46.806604 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:26:46.806678 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:26:46.806766 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:26:46.807769 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:46.808502 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:46.810294 (+ 1792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:46.811276 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:46.811445 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:46.814138 (+ 2693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:26:46.814144 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:26:46.814147 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:26:46.814190 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:26:46.817158 (+ 2968us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:26:46.818204 (+ 1046us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:26:46.818211 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:26:46.818215 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:26:46.818393 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:26:46.818872 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:26:46.818875 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:26:46.818877 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:26:46.819186 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:26:46.819324 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0810 03:26:46.820762 (+ 1438us) server_negotiation.cc:299] Negotiation successful
0810 03:26:46.821026 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260810 03:26:46.822348 7851 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786332336319544-28292-0/minicluster-data/ts-2/data/info.pb
I20260810 03:26:46.824518 8054 heartbeater.cc:347] Connected to a master server at 127.27.161.62:35965
I20260810 03:26:46.824793 8054 heartbeater.cc:464] Registering TS with master...
I20260810 03:26:46.825243 28292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iBH5/build/asan/bin/kudu as pid 7851
I20260810 03:26:46.825395 28292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786332336319544-28292-0/minicluster-data/ts-2/wal/instance
I20260810 03:26:46.825621 8054 heartbeater.cc:511] Master 127.27.161.62:35965 requested a full tablet report, sending...
I20260810 03:26:46.825851 28292 external_mini_cluster.cc:1428] Setting key 50a53c25d274999d984f9edbc87a8dff
I20260810 03:26:46.827172 7319 ts_manager.cc:194] Registered new tserver with Master: 9a1b431e2e804faab2004da408cd20f8 (127.27.161.3:40179)
I20260810 03:26:46.828118 7319 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.27.161.3@KRBTEST.COM'} at 127.27.161.3:40287
I20260810 03:26:46.828855 28292 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 10 03:26:46 dist-test-slave-vpvm krb5kdc[7252](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332406, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 10 03:26:46 dist-test-slave-vpvm krb5kdc[7252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.27.161.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 03:26:46 dist-test-slave-vpvm krb5kdc[7252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.27.161.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 03:26:47.268744 7844 heartbeater.cc:503] Master 127.27.161.62:35965 was elected leader, sending a full tablet report...
I20260810 03:26:47.830786 8054 heartbeater.cc:503] Master 127.27.161.62:35965 was elected leader, sending a full tablet report...
W20260810 03:26:47.894663 8066 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.006s user 0.003s sys 0.017s
W20260810 03:26:47.895926 7891 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:26:46.882609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33362 (local address 127.27.161.62:35965)
0810 03:26:46.882871 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0810 03:26:46.882878 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:26:46.883102 (+ 224us) server_negotiation.cc:407] Connection header received
0810 03:26:46.883315 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:26:46.883319 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:26:46.883399 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:26:46.883486 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:26:46.884960 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:46.885717 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:46.887073 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:46.887492 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:46.887609 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:47.895612 (+1008003us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33362: BlockingRecv error: recv got EOF from 127.0.0.1:33362 (error 108)
Metrics: {"server-negotiator.queue_time_us":101}
Aug 10 03:26:47 dist-test-slave-vpvm krb5kdc[7252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332406, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.27.161.62@KRBTEST.COM
I20260810 03:26:47.918038 7891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:47.900457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33372 (local address 127.27.161.62:35965)
0810 03:26:47.900665 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0810 03:26:47.900671 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:26:47.901059 (+ 388us) server_negotiation.cc:407] Connection header received
0810 03:26:47.901303 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:26:47.901308 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:26:47.901378 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:26:47.901470 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:26:47.903275 (+ 1805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:47.904028 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:47.905896 (+ 1868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:47.906393 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:47.906490 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:47.911358 (+ 4868us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:26:47.911367 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:26:47.911373 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:26:47.911427 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:26:47.915116 (+ 3689us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:26:47.915860 (+ 744us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:26:47.915866 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:26:47.915870 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:26:47.916027 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:26:47.916359 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:26:47.916364 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:26:47.916367 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:26:47.916788 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:26:47.917092 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0810 03:26:47.917361 (+ 269us) server_negotiation.cc:299] Negotiation successful
0810 03:26:47.917640 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260810 03:26:47.922483 7319 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33372:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 03:26:47.930259 7319 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 03:26:47.965564 8084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:47.948155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.3:40179 (local address 127.0.0.1:51190)
0810 03:26:47.948698 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0810 03:26:47.948713 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 03:26:47.948800 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:26:47.955459 (+ 6659us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:26:47.955467 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:26:47.955499 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:26:47.955976 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:26:47.955990 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:26:47.958168 (+ 2178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:26:47.958175 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:26:47.961902 (+ 3727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:26:47.961924 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:26:47.964485 (+ 2561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:26:47.964491 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:26:47.964730 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:47.964785 (+ 55us) client_negotiation.cc:769] Sending connection context
0810 03:26:47.964932 (+ 147us) client_negotiation.cc:240] Negotiation successful
0810 03:26:47.965157 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":302,"threads_started":1}
I20260810 03:26:47.968245 8082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:47.944761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.2:33609 (local address 127.0.0.1:36816)
0810 03:26:47.945550 (+ 789us) negotiation.cc:107] Waiting for socket to connect
0810 03:26:47.945589 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0810 03:26:47.945753 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:26:47.960035 (+ 14282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:26:47.960041 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:26:47.960076 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:26:47.960613 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:26:47.960625 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:26:47.962808 (+ 2183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:26:47.962812 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:26:47.965108 (+ 2296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:26:47.965127 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:26:47.967295 (+ 2168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:26:47.967302 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:26:47.967558 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:47.967594 (+ 36us) client_negotiation.cc:769] Sending connection context
0810 03:26:47.967687 (+ 93us) client_negotiation.cc:240] Negotiation successful
0810 03:26:47.967822 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":496,"mutex_wait_us":23,"thread_start_us":341,"threads_started":1}
I20260810 03:26:47.968309 8083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:47.944997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36816 (local address 127.27.161.2:33609)
0810 03:26:47.959300 (+ 14303us) server_negotiation.cc:206] Beginning negotiation
0810 03:26:47.959310 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 03:26:47.959346 (+ 36us) server_negotiation.cc:407] Connection header received
0810 03:26:47.959455 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:26:47.959464 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:26:47.959775 (+ 311us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:26:47.960850 (+ 1075us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:26:47.961107 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:47.962655 (+ 1548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:47.965358 (+ 2703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:47.967022 (+ 1664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:47.967509 (+ 487us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:47.967755 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0810 03:26:47.967834 (+ 79us) server_negotiation.cc:299] Negotiation successful
0810 03:26:47.968022 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14113,"thread_start_us":14196,"threads_started":1}
I20260810 03:26:47.969897 8085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:47.948472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51190 (local address 127.27.161.3:40179)
0810 03:26:47.954429 (+ 5957us) server_negotiation.cc:206] Beginning negotiation
0810 03:26:47.954438 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 03:26:47.954472 (+ 34us) server_negotiation.cc:407] Connection header received
0810 03:26:47.954582 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:26:47.954591 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:26:47.954907 (+ 316us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:26:47.955163 (+ 256us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:26:47.956252 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:47.957966 (+ 1714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:47.962188 (+ 4222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:47.963932 (+ 1744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:47.964074 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:47.964314 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0810 03:26:47.969383 (+ 5069us) server_negotiation.cc:299] Negotiation successful
0810 03:26:47.969598 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5790,"thread_start_us":5641,"threads_started":1}
I20260810 03:26:47.973055 7742 tablet_service.cc:1467] Processing CreateTablet for tablet 50c958667deb4d49bd1e31c73972777b (DEFAULT_TABLE table=test-table [id=a69294ec950d48d6a93f5329e1c33b12]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:26:47.973492 7946 tablet_service.cc:1467] Processing CreateTablet for tablet 50c958667deb4d49bd1e31c73972777b (DEFAULT_TABLE table=test-table [id=a69294ec950d48d6a93f5329e1c33b12]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:26:47.974921 7946 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 50c958667deb4d49bd1e31c73972777b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:26:47.974939 7742 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 50c958667deb4d49bd1e31c73972777b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:26:47.978624 8084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:47.955037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.1:35325 (local address 127.0.0.1:38524)
0810 03:26:47.966139 (+ 11102us) negotiation.cc:107] Waiting for socket to connect
0810 03:26:47.966160 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0810 03:26:47.966386 (+ 226us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:26:47.967352 (+ 966us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:26:47.967357 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:26:47.967385 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:26:47.969092 (+ 1707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:26:47.969104 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:26:47.971423 (+ 2319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:26:47.971429 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:26:47.973835 (+ 2406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:26:47.973870 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:26:47.977653 (+ 3783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:26:47.977661 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:26:47.977902 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:47.977939 (+ 37us) client_negotiation.cc:769] Sending connection context
0810 03:26:47.978067 (+ 128us) client_negotiation.cc:240] Negotiation successful
0810 03:26:47.978229 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10989,"mutex_wait_us":1,"thread_start_us":13669,"threads_started":1}
I20260810 03:26:47.979460 8087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:47.955693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38524 (local address 127.27.161.1:35325)
0810 03:26:47.959182 (+ 3489us) server_negotiation.cc:206] Beginning negotiation
0810 03:26:47.959188 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:26:47.966269 (+ 7081us) server_negotiation.cc:407] Connection header received
0810 03:26:47.966575 (+ 306us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:26:47.966585 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:26:47.966882 (+ 297us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:26:47.967072 (+ 190us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:26:47.969596 (+ 2524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:47.971227 (+ 1631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:47.974140 (+ 2913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:47.977140 (+ 3000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:47.977301 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:47.977533 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0810 03:26:47.978939 (+ 1406us) server_negotiation.cc:299] Negotiation successful
0810 03:26:47.979156 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3345,"thread_start_us":3208,"threads_started":1}
I20260810 03:26:47.983840 7542 tablet_service.cc:1467] Processing CreateTablet for tablet 50c958667deb4d49bd1e31c73972777b (DEFAULT_TABLE table=test-table [id=a69294ec950d48d6a93f5329e1c33b12]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:26:47.987290 7542 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 50c958667deb4d49bd1e31c73972777b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:26:47.997031 8089 tablet_bootstrap.cc:492] T 50c958667deb4d49bd1e31c73972777b P 9a1b431e2e804faab2004da408cd20f8: Bootstrap starting.
I20260810 03:26:47.997031 8088 tablet_bootstrap.cc:492] T 50c958667deb4d49bd1e31c73972777b P db6b5c27fe7b480f96dbf4e5afd7b6b9: Bootstrap starting.
I20260810 03:26:48.001034 8088 tablet_bootstrap.cc:654] T 50c958667deb4d49bd1e31c73972777b P db6b5c27fe7b480f96dbf4e5afd7b6b9: Neither blocks nor log segments found. Creating new log.
I20260810 03:26:48.003358 8089 tablet_bootstrap.cc:654] T 50c958667deb4d49bd1e31c73972777b P 9a1b431e2e804faab2004da408cd20f8: Neither blocks nor log segments found. Creating new log.
I20260810 03:26:48.003736 8088 log.cc:824] T 50c958667deb4d49bd1e31c73972777b P db6b5c27fe7b480f96dbf4e5afd7b6b9: Log is configured to *not* fsync() on all Append() calls
I20260810 03:26:48.004782 8089 log.cc:824] T 50c958667deb4d49bd1e31c73972777b P 9a1b431e2e804faab2004da408cd20f8: Log is configured to *not* fsync() on all Append() calls
I20260810 03:26:48.010160 8092 tablet_bootstrap.cc:492] T 50c958667deb4d49bd1e31c73972777b P 51a5b515913a4b10bd40695507e0fcca: Bootstrap starting.
I20260810 03:26:48.012017 8089 tablet_bootstrap.cc:492] T 50c958667deb4d49bd1e31c73972777b P 9a1b431e2e804faab2004da408cd20f8: No bootstrap required, opened a new log
I20260810 03:26:48.012408 8089 ts_tablet_manager.cc:1397] T 50c958667deb4d49bd1e31c73972777b P 9a1b431e2e804faab2004da408cd20f8: Time spent bootstrapping tablet: real 0.016s user 0.008s sys 0.004s
I20260810 03:26:48.014763 8088 tablet_bootstrap.cc:492] T 50c958667deb4d49bd1e31c73972777b P db6b5c27fe7b480f96dbf4e5afd7b6b9: No bootstrap required, opened a new log
I20260810 03:26:48.015084 8088 ts_tablet_manager.cc:1397] T 50c958667deb4d49bd1e31c73972777b P db6b5c27fe7b480f96dbf4e5afd7b6b9: Time spent bootstrapping tablet: real 0.018s user 0.007s sys 0.004s
I20260810 03:26:48.019826 8092 tablet_bootstrap.cc:654] T 50c958667deb4d49bd1e31c73972777b P 51a5b515913a4b10bd40695507e0fcca: Neither blocks nor log segments found. Creating new log.
I20260810 03:26:48.021397 8092 log.cc:824] T 50c958667deb4d49bd1e31c73972777b P 51a5b515913a4b10bd40695507e0fcca: Log is configured to *not* fsync() on all Append() calls
I20260810 03:26:48.023274 8089 raft_consensus.cc:348] T 50c958667deb4d49bd1e31c73972777b P 9a1b431e2e804faab2004da408cd20f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db6b5c27fe7b480f96dbf4e5afd7b6b9" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 33609 } } peers { permanent_uuid: "9a1b431e2e804faab2004da408cd20f8" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 40179 } } peers { permanent_uuid: "51a5b515913a4b10bd40695507e0fcca" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 35325 } }
I20260810 03:26:48.023918 8089 raft_consensus.cc:374] T 50c958667deb4d49bd1e31c73972777b P 9a1b431e2e804faab2004da408cd20f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:26:48.024080 8089 raft_consensus.cc:729] T 50c958667deb4d49bd1e31c73972777b P 9a1b431e2e804faab2004da408cd20f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a1b431e2e804faab2004da408cd20f8, State: Initialized, Role: FOLLOWER
I20260810 03:26:48.025112 8089 consensus_queue.cc:260] T 50c958667deb4d49bd1e31c73972777b P 9a1b431e2e804faab2004da408cd20f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db6b5c27fe7b480f96dbf4e5afd7b6b9" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 33609 } } peers { permanent_uuid: "9a1b431e2e804faab2004da408cd20f8" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 40179 } } peers { permanent_uuid: "51a5b515913a4b10bd40695507e0fcca" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 35325 } }
I20260810 03:26:48.026089 8088 raft_consensus.cc:348] T 50c958667deb4d49bd1e31c73972777b P db6b5c27fe7b480f96dbf4e5afd7b6b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db6b5c27fe7b480f96dbf4e5afd7b6b9" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 33609 } } peers { permanent_uuid: "9a1b431e2e804faab2004da408cd20f8" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 40179 } } peers { permanent_uuid: "51a5b515913a4b10bd40695507e0fcca" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 35325 } }
I20260810 03:26:48.026686 8088 raft_consensus.cc:374] T 50c958667deb4d49bd1e31c73972777b P db6b5c27fe7b480f96dbf4e5afd7b6b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:26:48.026829 8088 raft_consensus.cc:729] T 50c958667deb4d49bd1e31c73972777b P db6b5c27fe7b480f96dbf4e5afd7b6b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db6b5c27fe7b480f96dbf4e5afd7b6b9, State: Initialized, Role: FOLLOWER
I20260810 03:26:48.027719 8088 consensus_queue.cc:260] T 50c958667deb4d49bd1e31c73972777b P db6b5c27fe7b480f96dbf4e5afd7b6b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db6b5c27fe7b480f96dbf4e5afd7b6b9" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 33609 } } peers { permanent_uuid: "9a1b431e2e804faab2004da408cd20f8" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 40179 } } peers { permanent_uuid: "51a5b515913a4b10bd40695507e0fcca" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 35325 } }
I20260810 03:26:48.030331 8088 ts_tablet_manager.cc:1428] T 50c958667deb4d49bd1e31c73972777b P db6b5c27fe7b480f96dbf4e5afd7b6b9: Time spent starting tablet: real 0.015s user 0.011s sys 0.004s
I20260810 03:26:48.033221 8089 ts_tablet_manager.cc:1428] T 50c958667deb4d49bd1e31c73972777b P 9a1b431e2e804faab2004da408cd20f8: Time spent starting tablet: real 0.021s user 0.011s sys 0.008s
I20260810 03:26:48.036090 8092 tablet_bootstrap.cc:492] T 50c958667deb4d49bd1e31c73972777b P 51a5b515913a4b10bd40695507e0fcca: No bootstrap required, opened a new log
I20260810 03:26:48.036484 8092 ts_tablet_manager.cc:1397] T 50c958667deb4d49bd1e31c73972777b P 51a5b515913a4b10bd40695507e0fcca: Time spent bootstrapping tablet: real 0.027s user 0.006s sys 0.008s
I20260810 03:26:48.044188 8092 raft_consensus.cc:348] T 50c958667deb4d49bd1e31c73972777b P 51a5b515913a4b10bd40695507e0fcca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db6b5c27fe7b480f96dbf4e5afd7b6b9" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 33609 } } peers { permanent_uuid: "9a1b431e2e804faab2004da408cd20f8" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 40179 } } peers { permanent_uuid: "51a5b515913a4b10bd40695507e0fcca" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 35325 } }
I20260810 03:26:48.044633 8092 raft_consensus.cc:374] T 50c958667deb4d49bd1e31c73972777b P 51a5b515913a4b10bd40695507e0fcca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:26:48.044754 8092 raft_consensus.cc:729] T 50c958667deb4d49bd1e31c73972777b P 51a5b515913a4b10bd40695507e0fcca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51a5b515913a4b10bd40695507e0fcca, State: Initialized, Role: FOLLOWER
I20260810 03:26:48.045439 8092 consensus_queue.cc:260] T 50c958667deb4d49bd1e31c73972777b P 51a5b515913a4b10bd40695507e0fcca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db6b5c27fe7b480f96dbf4e5afd7b6b9" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 33609 } } peers { permanent_uuid: "9a1b431e2e804faab2004da408cd20f8" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 40179 } } peers { permanent_uuid: "51a5b515913a4b10bd40695507e0fcca" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 35325 } }
I20260810 03:26:48.045938 8098 raft_consensus.cc:482] T 50c958667deb4d49bd1e31c73972777b P db6b5c27fe7b480f96dbf4e5afd7b6b9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 03:26:48.046392 8098 raft_consensus.cc:504] T 50c958667deb4d49bd1e31c73972777b P db6b5c27fe7b480f96dbf4e5afd7b6b9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db6b5c27fe7b480f96dbf4e5afd7b6b9" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 33609 } } peers { permanent_uuid: "9a1b431e2e804faab2004da408cd20f8" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 40179 } } peers { permanent_uuid: "51a5b515913a4b10bd40695507e0fcca" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 35325 } }
I20260810 03:26:48.048324 8092 ts_tablet_manager.cc:1428] T 50c958667deb4d49bd1e31c73972777b P 51a5b515913a4b10bd40695507e0fcca: Time spent starting tablet: real 0.012s user 0.008s sys 0.003s
I20260810 03:26:48.054661 8098 leader_election.cc:288] T 50c958667deb4d49bd1e31c73972777b P db6b5c27fe7b480f96dbf4e5afd7b6b9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a1b431e2e804faab2004da408cd20f8 (127.27.161.3:40179), 51a5b515913a4b10bd40695507e0fcca (127.27.161.1:35325)
I20260810 03:26:48.061934 8087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:48.050698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.161.2:45215 (local address 127.27.161.1:35325)
0810 03:26:48.050894 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0810 03:26:48.050900 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:26:48.055536 (+ 4636us) server_negotiation.cc:407] Connection header received
0810 03:26:48.055701 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:26:48.055707 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:26:48.055783 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:26:48.055860 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:26:48.056544 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:48.057629 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:48.060191 (+ 2562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:48.061012 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:48.061188 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:48.061272 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0810 03:26:48.061548 (+ 276us) server_negotiation.cc:299] Negotiation successful
0810 03:26:48.061687 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260810 03:26:48.062052 8100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:48.049653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.1:35325 (local address 127.27.161.2:45215)
0810 03:26:48.055421 (+ 5768us) negotiation.cc:107] Waiting for socket to connect
0810 03:26:48.055436 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 03:26:48.055573 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:26:48.055903 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:26:48.055908 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:26:48.055932 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:26:48.056355 (+ 423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:26:48.056369 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:26:48.057786 (+ 1417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:26:48.057792 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:26:48.060026 (+ 2234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:26:48.060039 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:26:48.061225 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:26:48.061232 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:26:48.061385 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:48.061411 (+ 26us) client_negotiation.cc:769] Sending connection context
0810 03:26:48.061505 (+ 94us) client_negotiation.cc:240] Negotiation successful
0810 03:26:48.061668 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5624,"thread_start_us":4582,"threads_started":1}
I20260810 03:26:48.062428 8085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:48.048939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.161.2:46899 (local address 127.27.161.3:40179)
0810 03:26:48.049161 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0810 03:26:48.049168 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:26:48.054794 (+ 5626us) server_negotiation.cc:407] Connection header received
0810 03:26:48.055121 (+ 327us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:26:48.055127 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:26:48.055238 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:26:48.055302 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:26:48.056243 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:48.057083 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:48.059765 (+ 2682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:48.061084 (+ 1319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:48.061282 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:48.061364 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0810 03:26:48.062002 (+ 638us) server_negotiation.cc:299] Negotiation successful
0810 03:26:48.062157 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260810 03:26:48.063741 7570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50c958667deb4d49bd1e31c73972777b" candidate_uuid: "db6b5c27fe7b480f96dbf4e5afd7b6b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51a5b515913a4b10bd40695507e0fcca" is_pre_election: true
I20260810 03:26:48.063935 8101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:48.048840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.3:40179 (local address 127.27.161.2:46899)
0810 03:26:48.054666 (+ 5826us) negotiation.cc:107] Waiting for socket to connect
0810 03:26:48.054689 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0810 03:26:48.054940 (+ 251us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:26:48.055515 (+ 575us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:26:48.055520 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:26:48.055615 (+ 95us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:26:48.056045 (+ 430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:26:48.056064 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:26:48.057224 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:26:48.057230 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:26:48.059466 (+ 2236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:26:48.059512 (+ 46us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:26:48.061551 (+ 2039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:26:48.061559 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:26:48.061762 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:48.061798 (+ 36us) client_negotiation.cc:769] Sending connection context
0810 03:26:48.063239 (+ 1441us) client_negotiation.cc:240] Negotiation successful
0810 03:26:48.063410 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5665,"mutex_wait_us":1,"spinlock_wait_cycles":29056,"thread_start_us":6244,"threads_started":1}
I20260810 03:26:48.064355 7570 raft_consensus.cc:2454] T 50c958667deb4d49bd1e31c73972777b P 51a5b515913a4b10bd40695507e0fcca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db6b5c27fe7b480f96dbf4e5afd7b6b9 in term 0.
I20260810 03:26:48.065259 7675 leader_election.cc:302] T 50c958667deb4d49bd1e31c73972777b P db6b5c27fe7b480f96dbf4e5afd7b6b9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51a5b515913a4b10bd40695507e0fcca, db6b5c27fe7b480f96dbf4e5afd7b6b9; no voters:
I20260810 03:26:48.065191 7977 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50c958667deb4d49bd1e31c73972777b" candidate_uuid: "db6b5c27fe7b480f96dbf4e5afd7b6b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a1b431e2e804faab2004da408cd20f8" is_pre_election: true
I20260810 03:26:48.065721 8098 raft_consensus.cc:2781] T 50c958667deb4d49bd1e31c73972777b P db6b5c27fe7b480f96dbf4e5afd7b6b9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 03:26:48.065699 7977 raft_consensus.cc:2454] T 50c958667deb4d49bd1e31c73972777b P 9a1b431e2e804faab2004da408cd20f8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db6b5c27fe7b480f96dbf4e5afd7b6b9 in term 0.
I20260810 03:26:48.065882 8098 raft_consensus.cc:482] T 50c958667deb4d49bd1e31c73972777b P db6b5c27fe7b480f96dbf4e5afd7b6b9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 03:26:48.066020 8098 raft_consensus.cc:3037] T 50c958667deb4d49bd1e31c73972777b P db6b5c27fe7b480f96dbf4e5afd7b6b9 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:26:48.068364 8098 raft_consensus.cc:504] T 50c958667deb4d49bd1e31c73972777b P db6b5c27fe7b480f96dbf4e5afd7b6b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db6b5c27fe7b480f96dbf4e5afd7b6b9" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 33609 } } peers { permanent_uuid: "9a1b431e2e804faab2004da408cd20f8" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 40179 } } peers { permanent_uuid: "51a5b515913a4b10bd40695507e0fcca" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 35325 } }
I20260810 03:26:48.068995 8098 leader_election.cc:288] T 50c958667deb4d49bd1e31c73972777b P db6b5c27fe7b480f96dbf4e5afd7b6b9 [CANDIDATE]: Term 1 election: Requested vote from peers 9a1b431e2e804faab2004da408cd20f8 (127.27.161.3:40179), 51a5b515913a4b10bd40695507e0fcca (127.27.161.1:35325)
I20260810 03:26:48.069613 7977 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50c958667deb4d49bd1e31c73972777b" candidate_uuid: "db6b5c27fe7b480f96dbf4e5afd7b6b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a1b431e2e804faab2004da408cd20f8"
I20260810 03:26:48.069628 7570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50c958667deb4d49bd1e31c73972777b" candidate_uuid: "db6b5c27fe7b480f96dbf4e5afd7b6b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51a5b515913a4b10bd40695507e0fcca"
I20260810 03:26:48.069866 7570 raft_consensus.cc:3037] T 50c958667deb4d49bd1e31c73972777b P 51a5b515913a4b10bd40695507e0fcca [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:26:48.069856 7977 raft_consensus.cc:3037] T 50c958667deb4d49bd1e31c73972777b P 9a1b431e2e804faab2004da408cd20f8 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:26:48.073168 7570 raft_consensus.cc:2454] T 50c958667deb4d49bd1e31c73972777b P 51a5b515913a4b10bd40695507e0fcca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db6b5c27fe7b480f96dbf4e5afd7b6b9 in term 1.
I20260810 03:26:48.073161 7977 raft_consensus.cc:2454] T 50c958667deb4d49bd1e31c73972777b P 9a1b431e2e804faab2004da408cd20f8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db6b5c27fe7b480f96dbf4e5afd7b6b9 in term 1.
I20260810 03:26:48.073710 7675 leader_election.cc:302] T 50c958667deb4d49bd1e31c73972777b P db6b5c27fe7b480f96dbf4e5afd7b6b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51a5b515913a4b10bd40695507e0fcca, db6b5c27fe7b480f96dbf4e5afd7b6b9; no voters:
I20260810 03:26:48.074059 8098 raft_consensus.cc:2781] T 50c958667deb4d49bd1e31c73972777b P db6b5c27fe7b480f96dbf4e5afd7b6b9 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:26:48.074540 8098 raft_consensus.cc:686] T 50c958667deb4d49bd1e31c73972777b P db6b5c27fe7b480f96dbf4e5afd7b6b9 [term 1 LEADER]: Becoming Leader. State: Replica: db6b5c27fe7b480f96dbf4e5afd7b6b9, State: Running, Role: LEADER
W20260810 03:26:48.075073 8056 tablet.cc:2369] T 50c958667deb4d49bd1e31c73972777b P 9a1b431e2e804faab2004da408cd20f8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 03:26:48.075183 8098 consensus_queue.cc:237] T 50c958667deb4d49bd1e31c73972777b P db6b5c27fe7b480f96dbf4e5afd7b6b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db6b5c27fe7b480f96dbf4e5afd7b6b9" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 33609 } } peers { permanent_uuid: "9a1b431e2e804faab2004da408cd20f8" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 40179 } } peers { permanent_uuid: "51a5b515913a4b10bd40695507e0fcca" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 35325 } }
I20260810 03:26:48.080855 7320 catalog_manager.cc:5495] T 50c958667deb4d49bd1e31c73972777b P db6b5c27fe7b480f96dbf4e5afd7b6b9 reported cstate change: term changed from 0 to 1, leader changed from <none> to db6b5c27fe7b480f96dbf4e5afd7b6b9 (127.27.161.2). New cstate: current_term: 1 leader_uuid: "db6b5c27fe7b480f96dbf4e5afd7b6b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db6b5c27fe7b480f96dbf4e5afd7b6b9" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 33609 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9a1b431e2e804faab2004da408cd20f8" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 40179 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "51a5b515913a4b10bd40695507e0fcca" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 35325 } health_report { overall_health: UNKNOWN } } }
I20260810 03:26:48.117183 8083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:48.111434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36826 (local address 127.27.161.2:33609)
0810 03:26:48.111658 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0810 03:26:48.111663 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:26:48.111723 (+ 60us) server_negotiation.cc:407] Connection header received
0810 03:26:48.112059 (+ 336us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:26:48.112065 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:26:48.112178 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:26:48.112316 (+ 138us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 03:26:48.112652 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:48.113450 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:48.114987 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:48.115415 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:48.115546 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:48.115990 (+ 444us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 03:26:48.116564 (+ 574us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 03:26:48.116698 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0810 03:26:48.116811 (+ 113us) server_negotiation.cc:299] Negotiation successful
0810 03:26:48.116901 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260810 03:26:48.126416 7977 raft_consensus.cc:1261] T 50c958667deb4d49bd1e31c73972777b P 9a1b431e2e804faab2004da408cd20f8 [term 1 FOLLOWER]: Refusing update from remote peer db6b5c27fe7b480f96dbf4e5afd7b6b9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:26:48.126420 7570 raft_consensus.cc:1261] T 50c958667deb4d49bd1e31c73972777b P 51a5b515913a4b10bd40695507e0fcca [term 1 FOLLOWER]: Refusing update from remote peer db6b5c27fe7b480f96dbf4e5afd7b6b9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:26:48.127660 8098 consensus_queue.cc:1035] T 50c958667deb4d49bd1e31c73972777b P db6b5c27fe7b480f96dbf4e5afd7b6b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51a5b515913a4b10bd40695507e0fcca" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 35325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:26:48.127992 8103 consensus_queue.cc:1035] T 50c958667deb4d49bd1e31c73972777b P db6b5c27fe7b480f96dbf4e5afd7b6b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a1b431e2e804faab2004da408cd20f8" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 40179 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:26:48.143491 8114 mvcc.cc:205] Tried to move back new op lower bound from 7316817543673499648 to 7316817543480987648. Current Snapshot: MvccSnapshot[applied={T|T < 7316817543673499648}]
I20260810 03:26:48.159675 7319 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33372:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 03:26:48.160028 7319 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33372:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 03:26:48.164587 7319 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6882776d888e4399b85872d4661cecf3: Sending DeleteTablet for 3 replicas of tablet 50c958667deb4d49bd1e31c73972777b
I20260810 03:26:48.166108 7542 tablet_service.cc:1514] Processing DeleteTablet for tablet 50c958667deb4d49bd1e31c73972777b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:26:48 UTC) from {username='oryx', principal='oryx/127.27.161.62@KRBTEST.COM'} at 127.0.0.1:38524
I20260810 03:26:48.166392 7946 tablet_service.cc:1514] Processing DeleteTablet for tablet 50c958667deb4d49bd1e31c73972777b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:26:48 UTC) from {username='oryx', principal='oryx/127.27.161.62@KRBTEST.COM'} at 127.0.0.1:51190
I20260810 03:26:48.166435 7742 tablet_service.cc:1514] Processing DeleteTablet for tablet 50c958667deb4d49bd1e31c73972777b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:26:48 UTC) from {username='oryx', principal='oryx/127.27.161.62@KRBTEST.COM'} at 127.0.0.1:36816
I20260810 03:26:48.169514 7319 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33372:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 03:26:48.171337 7319 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 03:26:48.174043 8123 tablet_replica.cc:330] T 50c958667deb4d49bd1e31c73972777b P 9a1b431e2e804faab2004da408cd20f8: stopping tablet replica
I20260810 03:26:48.174212 8122 tablet_replica.cc:330] T 50c958667deb4d49bd1e31c73972777b P 51a5b515913a4b10bd40695507e0fcca: stopping tablet replica
I20260810 03:26:48.174813 8122 raft_consensus.cc:2229] T 50c958667deb4d49bd1e31c73972777b P 51a5b515913a4b10bd40695507e0fcca [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 03:26:48.175453 8122 raft_consensus.cc:2258] T 50c958667deb4d49bd1e31c73972777b P 51a5b515913a4b10bd40695507e0fcca [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:26:48.176124 8123 raft_consensus.cc:2229] T 50c958667deb4d49bd1e31c73972777b P 9a1b431e2e804faab2004da408cd20f8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 03:26:48.177140 8124 tablet_replica.cc:330] T 50c958667deb4d49bd1e31c73972777b P db6b5c27fe7b480f96dbf4e5afd7b6b9: stopping tablet replica
I20260810 03:26:48.177660 8124 raft_consensus.cc:2229] T 50c958667deb4d49bd1e31c73972777b P db6b5c27fe7b480f96dbf4e5afd7b6b9 [term 1 LEADER]: Raft consensus shutting down.
I20260810 03:26:48.178509 8124 raft_consensus.cc:2258] T 50c958667deb4d49bd1e31c73972777b P db6b5c27fe7b480f96dbf4e5afd7b6b9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:26:48.179458 8123 raft_consensus.cc:2258] T 50c958667deb4d49bd1e31c73972777b P 9a1b431e2e804faab2004da408cd20f8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:26:48.180580 8122 ts_tablet_manager.cc:1905] T 50c958667deb4d49bd1e31c73972777b P 51a5b515913a4b10bd40695507e0fcca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:26:48.181443 8124 ts_tablet_manager.cc:1905] T 50c958667deb4d49bd1e31c73972777b P db6b5c27fe7b480f96dbf4e5afd7b6b9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:26:48.182495 8123 ts_tablet_manager.cc:1905] T 50c958667deb4d49bd1e31c73972777b P 9a1b431e2e804faab2004da408cd20f8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:26:48.182587 7946 tablet_service.cc:1467] Processing CreateTablet for tablet a281469115a94ba281a7bf9772255ebc (DEFAULT_TABLE table=test-table [id=c892b1cc7de74c9a862af6065f116311]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:26:48.183084 7946 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a281469115a94ba281a7bf9772255ebc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:26:48.183905 7542 tablet_service.cc:1467] Processing CreateTablet for tablet a281469115a94ba281a7bf9772255ebc (DEFAULT_TABLE table=test-table [id=c892b1cc7de74c9a862af6065f116311]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:26:48.184434 7542 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a281469115a94ba281a7bf9772255ebc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:26:48.187018 7742 tablet_service.cc:1467] Processing CreateTablet for tablet a281469115a94ba281a7bf9772255ebc (DEFAULT_TABLE table=test-table [id=c892b1cc7de74c9a862af6065f116311]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:26:48.187566 7742 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a281469115a94ba281a7bf9772255ebc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:26:48.188405 8122 ts_tablet_manager.cc:1918] T 50c958667deb4d49bd1e31c73972777b P 51a5b515913a4b10bd40695507e0fcca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 03:26:48.188651 8122 log.cc:1196] T 50c958667deb4d49bd1e31c73972777b P 51a5b515913a4b10bd40695507e0fcca: Deleting WAL directory at /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786332336319544-28292-0/minicluster-data/ts-0/wal/wals/50c958667deb4d49bd1e31c73972777b
I20260810 03:26:48.189302 8122 ts_tablet_manager.cc:1939] T 50c958667deb4d49bd1e31c73972777b P 51a5b515913a4b10bd40695507e0fcca: Deleting consensus metadata
I20260810 03:26:48.191699 7294 catalog_manager.cc:4841] TS 51a5b515913a4b10bd40695507e0fcca (127.27.161.1:35325): tablet 50c958667deb4d49bd1e31c73972777b (table test-table [id=a69294ec950d48d6a93f5329e1c33b12]) successfully deleted
I20260810 03:26:48.191838 8092 tablet_bootstrap.cc:492] T a281469115a94ba281a7bf9772255ebc P 51a5b515913a4b10bd40695507e0fcca: Bootstrap starting.
I20260810 03:26:48.193569 8092 tablet_bootstrap.cc:654] T a281469115a94ba281a7bf9772255ebc P 51a5b515913a4b10bd40695507e0fcca: Neither blocks nor log segments found. Creating new log.
I20260810 03:26:48.193874 8089 tablet_bootstrap.cc:492] T a281469115a94ba281a7bf9772255ebc P 9a1b431e2e804faab2004da408cd20f8: Bootstrap starting.
I20260810 03:26:48.194581 8088 tablet_bootstrap.cc:492] T a281469115a94ba281a7bf9772255ebc P db6b5c27fe7b480f96dbf4e5afd7b6b9: Bootstrap starting.
I20260810 03:26:48.195716 8089 tablet_bootstrap.cc:654] T a281469115a94ba281a7bf9772255ebc P 9a1b431e2e804faab2004da408cd20f8: Neither blocks nor log segments found. Creating new log.
I20260810 03:26:48.196419 8088 tablet_bootstrap.cc:654] T a281469115a94ba281a7bf9772255ebc P db6b5c27fe7b480f96dbf4e5afd7b6b9: Neither blocks nor log segments found. Creating new log.
I20260810 03:26:48.200304 8123 ts_tablet_manager.cc:1918] T 50c958667deb4d49bd1e31c73972777b P 9a1b431e2e804faab2004da408cd20f8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 03:26:48.200659 8123 log.cc:1196] T 50c958667deb4d49bd1e31c73972777b P 9a1b431e2e804faab2004da408cd20f8: Deleting WAL directory at /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786332336319544-28292-0/minicluster-data/ts-2/wal/wals/50c958667deb4d49bd1e31c73972777b
I20260810 03:26:48.201421 8123 ts_tablet_manager.cc:1939] T 50c958667deb4d49bd1e31c73972777b P 9a1b431e2e804faab2004da408cd20f8: Deleting consensus metadata
I20260810 03:26:48.203984 7293 catalog_manager.cc:4841] TS 9a1b431e2e804faab2004da408cd20f8 (127.27.161.3:40179): tablet 50c958667deb4d49bd1e31c73972777b (table test-table [id=a69294ec950d48d6a93f5329e1c33b12]) successfully deleted
I20260810 03:26:48.204411 8088 tablet_bootstrap.cc:492] T a281469115a94ba281a7bf9772255ebc P db6b5c27fe7b480f96dbf4e5afd7b6b9: No bootstrap required, opened a new log
I20260810 03:26:48.204674 8088 ts_tablet_manager.cc:1397] T a281469115a94ba281a7bf9772255ebc P db6b5c27fe7b480f96dbf4e5afd7b6b9: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.001s
I20260810 03:26:48.206580 8124 ts_tablet_manager.cc:1918] T 50c958667deb4d49bd1e31c73972777b P db6b5c27fe7b480f96dbf4e5afd7b6b9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 03:26:48.206888 8124 log.cc:1196] T 50c958667deb4d49bd1e31c73972777b P db6b5c27fe7b480f96dbf4e5afd7b6b9: Deleting WAL directory at /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786332336319544-28292-0/minicluster-data/ts-1/wal/wals/50c958667deb4d49bd1e31c73972777b
I20260810 03:26:48.207708 8124 ts_tablet_manager.cc:1939] T 50c958667deb4d49bd1e31c73972777b P db6b5c27fe7b480f96dbf4e5afd7b6b9: Deleting consensus metadata
I20260810 03:26:48.207544 8088 raft_consensus.cc:348] T a281469115a94ba281a7bf9772255ebc P db6b5c27fe7b480f96dbf4e5afd7b6b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51a5b515913a4b10bd40695507e0fcca" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 35325 } } peers { permanent_uuid: "db6b5c27fe7b480f96dbf4e5afd7b6b9" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 33609 } } peers { permanent_uuid: "9a1b431e2e804faab2004da408cd20f8" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 40179 } }
I20260810 03:26:48.208037 8088 raft_consensus.cc:374] T a281469115a94ba281a7bf9772255ebc P db6b5c27fe7b480f96dbf4e5afd7b6b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:26:48.208199 8088 raft_consensus.cc:729] T a281469115a94ba281a7bf9772255ebc P db6b5c27fe7b480f96dbf4e5afd7b6b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db6b5c27fe7b480f96dbf4e5afd7b6b9, State: Initialized, Role: FOLLOWER
I20260810 03:26:48.208686 8088 consensus_queue.cc:260] T a281469115a94ba281a7bf9772255ebc P db6b5c27fe7b480f96dbf4e5afd7b6b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51a5b515913a4b10bd40695507e0fcca" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 35325 } } peers { permanent_uuid: "db6b5c27fe7b480f96dbf4e5afd7b6b9" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 33609 } } peers { permanent_uuid: "9a1b431e2e804faab2004da408cd20f8" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 40179 } }
I20260810 03:26:48.210935 7293 catalog_manager.cc:4841] TS db6b5c27fe7b480f96dbf4e5afd7b6b9 (127.27.161.2:33609): tablet 50c958667deb4d49bd1e31c73972777b (table test-table [id=a69294ec950d48d6a93f5329e1c33b12]) successfully deleted
I20260810 03:26:48.211812 8088 ts_tablet_manager.cc:1428] T a281469115a94ba281a7bf9772255ebc P db6b5c27fe7b480f96dbf4e5afd7b6b9: Time spent starting tablet: real 0.007s user 0.002s sys 0.001s
I20260810 03:26:48.212759 8092 tablet_bootstrap.cc:492] T a281469115a94ba281a7bf9772255ebc P 51a5b515913a4b10bd40695507e0fcca: No bootstrap required, opened a new log
I20260810 03:26:48.212780 8089 tablet_bootstrap.cc:492] T a281469115a94ba281a7bf9772255ebc P 9a1b431e2e804faab2004da408cd20f8: No bootstrap required, opened a new log
I20260810 03:26:48.212944 8089 ts_tablet_manager.cc:1397] T a281469115a94ba281a7bf9772255ebc P 9a1b431e2e804faab2004da408cd20f8: Time spent bootstrapping tablet: real 0.019s user 0.002s sys 0.014s
I20260810 03:26:48.213006 8092 ts_tablet_manager.cc:1397] T a281469115a94ba281a7bf9772255ebc P 51a5b515913a4b10bd40695507e0fcca: Time spent bootstrapping tablet: real 0.021s user 0.004s sys 0.001s
I20260810 03:26:48.214025 8089 raft_consensus.cc:348] T a281469115a94ba281a7bf9772255ebc P 9a1b431e2e804faab2004da408cd20f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51a5b515913a4b10bd40695507e0fcca" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 35325 } } peers { permanent_uuid: "db6b5c27fe7b480f96dbf4e5afd7b6b9" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 33609 } } peers { permanent_uuid: "9a1b431e2e804faab2004da408cd20f8" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 40179 } }
I20260810 03:26:48.214030 8092 raft_consensus.cc:348] T a281469115a94ba281a7bf9772255ebc P 51a5b515913a4b10bd40695507e0fcca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51a5b515913a4b10bd40695507e0fcca" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 35325 } } peers { permanent_uuid: "db6b5c27fe7b480f96dbf4e5afd7b6b9" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 33609 } } peers { permanent_uuid: "9a1b431e2e804faab2004da408cd20f8" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 40179 } }
I20260810 03:26:48.214293 8092 raft_consensus.cc:374] T a281469115a94ba281a7bf9772255ebc P 51a5b515913a4b10bd40695507e0fcca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:26:48.214309 8089 raft_consensus.cc:374] T a281469115a94ba281a7bf9772255ebc P 9a1b431e2e804faab2004da408cd20f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:26:48.214430 8092 raft_consensus.cc:729] T a281469115a94ba281a7bf9772255ebc P 51a5b515913a4b10bd40695507e0fcca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51a5b515913a4b10bd40695507e0fcca, State: Initialized, Role: FOLLOWER
I20260810 03:26:48.214453 8089 raft_consensus.cc:729] T a281469115a94ba281a7bf9772255ebc P 9a1b431e2e804faab2004da408cd20f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a1b431e2e804faab2004da408cd20f8, State: Initialized, Role: FOLLOWER
I20260810 03:26:48.214821 8089 consensus_queue.cc:260] T a281469115a94ba281a7bf9772255ebc P 9a1b431e2e804faab2004da408cd20f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51a5b515913a4b10bd40695507e0fcca" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 35325 } } peers { permanent_uuid: "db6b5c27fe7b480f96dbf4e5afd7b6b9" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 33609 } } peers { permanent_uuid: "9a1b431e2e804faab2004da408cd20f8" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 40179 } }
I20260810 03:26:48.214810 8092 consensus_queue.cc:260] T a281469115a94ba281a7bf9772255ebc P 51a5b515913a4b10bd40695507e0fcca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51a5b515913a4b10bd40695507e0fcca" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 35325 } } peers { permanent_uuid: "db6b5c27fe7b480f96dbf4e5afd7b6b9" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 33609 } } peers { permanent_uuid: "9a1b431e2e804faab2004da408cd20f8" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 40179 } }
I20260810 03:26:48.215480 8092 ts_tablet_manager.cc:1428] T a281469115a94ba281a7bf9772255ebc P 51a5b515913a4b10bd40695507e0fcca: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260810 03:26:48.216123 8089 ts_tablet_manager.cc:1428] T a281469115a94ba281a7bf9772255ebc P 9a1b431e2e804faab2004da408cd20f8: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
W20260810 03:26:48.263427 7847 tablet.cc:2369] T a281469115a94ba281a7bf9772255ebc P db6b5c27fe7b480f96dbf4e5afd7b6b9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 03:26:48.295841 8097 raft_consensus.cc:482] T a281469115a94ba281a7bf9772255ebc P 9a1b431e2e804faab2004da408cd20f8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 03:26:48.296082 8097 raft_consensus.cc:504] T a281469115a94ba281a7bf9772255ebc P 9a1b431e2e804faab2004da408cd20f8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51a5b515913a4b10bd40695507e0fcca" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 35325 } } peers { permanent_uuid: "db6b5c27fe7b480f96dbf4e5afd7b6b9" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 33609 } } peers { permanent_uuid: "9a1b431e2e804faab2004da408cd20f8" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 40179 } }
I20260810 03:26:48.297856 8097 leader_election.cc:288] T a281469115a94ba281a7bf9772255ebc P 9a1b431e2e804faab2004da408cd20f8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51a5b515913a4b10bd40695507e0fcca (127.27.161.1:35325), db6b5c27fe7b480f96dbf4e5afd7b6b9 (127.27.161.2:33609)
I20260810 03:26:48.306704 8083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:48.298350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.161.3:35099 (local address 127.27.161.2:33609)
0810 03:26:48.298571 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0810 03:26:48.298577 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:26:48.299890 (+ 1313us) server_negotiation.cc:407] Connection header received
0810 03:26:48.300168 (+ 278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:26:48.300176 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:26:48.300263 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:26:48.300338 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:26:48.301300 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:48.302540 (+ 1240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:48.304805 (+ 2265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:48.305644 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:48.305803 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:48.305888 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0810 03:26:48.306236 (+ 348us) server_negotiation.cc:299] Negotiation successful
0810 03:26:48.306376 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260810 03:26:48.306754 8130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:48.298841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.2:33609 (local address 127.27.161.3:35099)
0810 03:26:48.299754 (+ 913us) negotiation.cc:107] Waiting for socket to connect
0810 03:26:48.299776 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0810 03:26:48.299943 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:26:48.300566 (+ 623us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:26:48.300572 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:26:48.300600 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:26:48.301061 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:26:48.301077 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:26:48.302721 (+ 1644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:26:48.302726 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:26:48.304652 (+ 1926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:26:48.304663 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:26:48.305874 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:26:48.305880 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:26:48.306070 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:48.306098 (+ 28us) client_negotiation.cc:769] Sending connection context
0810 03:26:48.306188 (+ 90us) client_negotiation.cc:240] Negotiation successful
0810 03:26:48.306338 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":771,"mutex_wait_us":19,"thread_start_us":529,"threads_started":1}
I20260810 03:26:48.307212 8087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:48.298350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.161.3:46975 (local address 127.27.161.1:35325)
0810 03:26:48.298547 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0810 03:26:48.298554 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:26:48.299552 (+ 998us) server_negotiation.cc:407] Connection header received
0810 03:26:48.299837 (+ 285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:26:48.299842 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:26:48.299945 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:26:48.300029 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:26:48.300561 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:48.301563 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:48.304358 (+ 2795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:48.305732 (+ 1374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:48.305935 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:48.306071 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0810 03:26:48.306716 (+ 645us) server_negotiation.cc:299] Negotiation successful
0810 03:26:48.306871 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260810 03:26:48.307884 7779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a281469115a94ba281a7bf9772255ebc" candidate_uuid: "9a1b431e2e804faab2004da408cd20f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db6b5c27fe7b480f96dbf4e5afd7b6b9" is_pre_election: true
I20260810 03:26:48.308112 8129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:48.298525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.1:35325 (local address 127.27.161.3:46975)
0810 03:26:48.299395 (+ 870us) negotiation.cc:107] Waiting for socket to connect
0810 03:26:48.299417 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0810 03:26:48.299677 (+ 260us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:26:48.300074 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:26:48.300079 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:26:48.300108 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:26:48.300407 (+ 299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:26:48.300416 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:26:48.301735 (+ 1319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:26:48.301740 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:26:48.304172 (+ 2432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:26:48.304189 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:26:48.306264 (+ 2075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:26:48.306271 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:26:48.306484 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:48.306524 (+ 40us) client_negotiation.cc:769] Sending connection context
0810 03:26:48.307498 (+ 974us) client_negotiation.cc:240] Negotiation successful
0810 03:26:48.307661 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":638,"mutex_wait_us":26,"thread_start_us":418,"threads_started":1}
I20260810 03:26:48.308352 7779 raft_consensus.cc:2454] T a281469115a94ba281a7bf9772255ebc P db6b5c27fe7b480f96dbf4e5afd7b6b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a1b431e2e804faab2004da408cd20f8 in term 0.
I20260810 03:26:48.308786 7570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a281469115a94ba281a7bf9772255ebc" candidate_uuid: "9a1b431e2e804faab2004da408cd20f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51a5b515913a4b10bd40695507e0fcca" is_pre_election: true
I20260810 03:26:48.309051 7570 raft_consensus.cc:2454] T a281469115a94ba281a7bf9772255ebc P 51a5b515913a4b10bd40695507e0fcca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a1b431e2e804faab2004da408cd20f8 in term 0.
I20260810 03:26:48.309104 7877 leader_election.cc:302] T a281469115a94ba281a7bf9772255ebc P 9a1b431e2e804faab2004da408cd20f8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a1b431e2e804faab2004da408cd20f8, db6b5c27fe7b480f96dbf4e5afd7b6b9; no voters:
I20260810 03:26:48.309571 8097 raft_consensus.cc:2781] T a281469115a94ba281a7bf9772255ebc P 9a1b431e2e804faab2004da408cd20f8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 03:26:48.309692 8097 raft_consensus.cc:482] T a281469115a94ba281a7bf9772255ebc P 9a1b431e2e804faab2004da408cd20f8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 03:26:48.309794 8097 raft_consensus.cc:3037] T a281469115a94ba281a7bf9772255ebc P 9a1b431e2e804faab2004da408cd20f8 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:26:48.312229 8097 raft_consensus.cc:504] T a281469115a94ba281a7bf9772255ebc P 9a1b431e2e804faab2004da408cd20f8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51a5b515913a4b10bd40695507e0fcca" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 35325 } } peers { permanent_uuid: "db6b5c27fe7b480f96dbf4e5afd7b6b9" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 33609 } } peers { permanent_uuid: "9a1b431e2e804faab2004da408cd20f8" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 40179 } }
I20260810 03:26:48.312989 8097 leader_election.cc:288] T a281469115a94ba281a7bf9772255ebc P 9a1b431e2e804faab2004da408cd20f8 [CANDIDATE]: Term 1 election: Requested vote from peers 51a5b515913a4b10bd40695507e0fcca (127.27.161.1:35325), db6b5c27fe7b480f96dbf4e5afd7b6b9 (127.27.161.2:33609)
I20260810 03:26:48.313637 7779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a281469115a94ba281a7bf9772255ebc" candidate_uuid: "9a1b431e2e804faab2004da408cd20f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db6b5c27fe7b480f96dbf4e5afd7b6b9"
I20260810 03:26:48.313679 7570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a281469115a94ba281a7bf9772255ebc" candidate_uuid: "9a1b431e2e804faab2004da408cd20f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51a5b515913a4b10bd40695507e0fcca"
I20260810 03:26:48.313920 7779 raft_consensus.cc:3037] T a281469115a94ba281a7bf9772255ebc P db6b5c27fe7b480f96dbf4e5afd7b6b9 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:26:48.313941 7570 raft_consensus.cc:3037] T a281469115a94ba281a7bf9772255ebc P 51a5b515913a4b10bd40695507e0fcca [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:26:48.316745 7779 raft_consensus.cc:2454] T a281469115a94ba281a7bf9772255ebc P db6b5c27fe7b480f96dbf4e5afd7b6b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a1b431e2e804faab2004da408cd20f8 in term 1.
I20260810 03:26:48.317209 7570 raft_consensus.cc:2454] T a281469115a94ba281a7bf9772255ebc P 51a5b515913a4b10bd40695507e0fcca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a1b431e2e804faab2004da408cd20f8 in term 1.
I20260810 03:26:48.317284 7877 leader_election.cc:302] T a281469115a94ba281a7bf9772255ebc P 9a1b431e2e804faab2004da408cd20f8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a1b431e2e804faab2004da408cd20f8, db6b5c27fe7b480f96dbf4e5afd7b6b9; no voters:
I20260810 03:26:48.317611 8097 raft_consensus.cc:2781] T a281469115a94ba281a7bf9772255ebc P 9a1b431e2e804faab2004da408cd20f8 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:26:48.318161 8097 raft_consensus.cc:686] T a281469115a94ba281a7bf9772255ebc P 9a1b431e2e804faab2004da408cd20f8 [term 1 LEADER]: Becoming Leader. State: Replica: 9a1b431e2e804faab2004da408cd20f8, State: Running, Role: LEADER
I20260810 03:26:48.318668 8097 consensus_queue.cc:237] T a281469115a94ba281a7bf9772255ebc P 9a1b431e2e804faab2004da408cd20f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51a5b515913a4b10bd40695507e0fcca" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 35325 } } peers { permanent_uuid: "db6b5c27fe7b480f96dbf4e5afd7b6b9" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 33609 } } peers { permanent_uuid: "9a1b431e2e804faab2004da408cd20f8" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 40179 } }
I20260810 03:26:48.322535 7318 catalog_manager.cc:5495] T a281469115a94ba281a7bf9772255ebc P 9a1b431e2e804faab2004da408cd20f8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9a1b431e2e804faab2004da408cd20f8 (127.27.161.3). New cstate: current_term: 1 leader_uuid: "9a1b431e2e804faab2004da408cd20f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51a5b515913a4b10bd40695507e0fcca" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 35325 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "db6b5c27fe7b480f96dbf4e5afd7b6b9" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 33609 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9a1b431e2e804faab2004da408cd20f8" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 40179 } health_report { overall_health: HEALTHY } } }
Aug 10 03:26:48 dist-test-slave-vpvm krb5kdc[7252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332404, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.161.62@KRBTEST.COM for oryx/127.27.161.62@KRBTEST.COM
I20260810 03:26:48.373308 8141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:48.356965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:35965 (local address 127.0.0.1:33374)
0810 03:26:48.357775 (+ 810us) negotiation.cc:107] Waiting for socket to connect
0810 03:26:48.357794 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 03:26:48.357992 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:26:48.358352 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:26:48.358355 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:26:48.359143 (+ 788us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:26:48.359437 (+ 294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:26:48.359447 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:26:48.360897 (+ 1450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:26:48.360901 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:26:48.361861 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:26:48.361872 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:26:48.362901 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:26:48.362906 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:26:48.363047 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:48.363898 (+ 851us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:26:48.363917 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:26:48.366487 (+ 2570us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:26:48.370643 (+ 4156us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:26:48.370661 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:26:48.370668 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:26:48.371194 (+ 526us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:26:48.371634 (+ 440us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:26:48.371638 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:26:48.371641 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:26:48.371798 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:26:48.372423 (+ 625us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:26:48.372430 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:26:48.372600 (+ 170us) client_negotiation.cc:769] Sending connection context
0810 03:26:48.372745 (+ 145us) client_negotiation.cc:240] Negotiation successful
0810 03:26:48.372974 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":645,"thread_start_us":455,"threads_started":1}
I20260810 03:26:48.374110 7891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:48.356997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33374 (local address 127.27.161.62:35965)
0810 03:26:48.357280 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0810 03:26:48.357291 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 03:26:48.357895 (+ 604us) server_negotiation.cc:407] Connection header received
0810 03:26:48.358122 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:26:48.358128 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:26:48.358211 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:26:48.358334 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:26:48.359600 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:48.360745 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:48.362041 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:48.362691 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:48.362821 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:48.366731 (+ 3910us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:26:48.366739 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:26:48.366744 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:26:48.366786 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:26:48.370464 (+ 3678us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:26:48.371330 (+ 866us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:26:48.371336 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:26:48.371340 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:26:48.371493 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:26:48.371914 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:26:48.371918 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:26:48.371922 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:26:48.372279 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:26:48.372422 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0810 03:26:48.372892 (+ 470us) server_negotiation.cc:299] Negotiation successful
0810 03:26:48.373136 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126}
I20260810 03:26:48.379407 7320 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.27.161.62@KRBTEST.COM'} at 127.0.0.1:33374:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260810 03:26:48.380590 7320 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 03:26:48.388130 7542 tablet_service.cc:1467] Processing CreateTablet for tablet 0df834d8e3484e3fa617e27fa69f808a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c3417753279d42acb68ff1dbf7c96f55]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 03:26:48.388288 7742 tablet_service.cc:1467] Processing CreateTablet for tablet 0df834d8e3484e3fa617e27fa69f808a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c3417753279d42acb68ff1dbf7c96f55]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 03:26:48.388686 7946 tablet_service.cc:1467] Processing CreateTablet for tablet 0df834d8e3484e3fa617e27fa69f808a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c3417753279d42acb68ff1dbf7c96f55]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 03:26:48.388875 7542 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0df834d8e3484e3fa617e27fa69f808a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:26:48.388913 7742 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0df834d8e3484e3fa617e27fa69f808a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:26:48.389206 7946 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0df834d8e3484e3fa617e27fa69f808a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:26:48.396888 8088 tablet_bootstrap.cc:492] T 0df834d8e3484e3fa617e27fa69f808a P db6b5c27fe7b480f96dbf4e5afd7b6b9: Bootstrap starting.
I20260810 03:26:48.397158 8092 tablet_bootstrap.cc:492] T 0df834d8e3484e3fa617e27fa69f808a P 51a5b515913a4b10bd40695507e0fcca: Bootstrap starting.
I20260810 03:26:48.398962 8092 tablet_bootstrap.cc:654] T 0df834d8e3484e3fa617e27fa69f808a P 51a5b515913a4b10bd40695507e0fcca: Neither blocks nor log segments found. Creating new log.
I20260810 03:26:48.399001 8088 tablet_bootstrap.cc:654] T 0df834d8e3484e3fa617e27fa69f808a P db6b5c27fe7b480f96dbf4e5afd7b6b9: Neither blocks nor log segments found. Creating new log.
I20260810 03:26:48.401477 8089 tablet_bootstrap.cc:492] T 0df834d8e3484e3fa617e27fa69f808a P 9a1b431e2e804faab2004da408cd20f8: Bootstrap starting.
I20260810 03:26:48.402915 8088 tablet_bootstrap.cc:492] T 0df834d8e3484e3fa617e27fa69f808a P db6b5c27fe7b480f96dbf4e5afd7b6b9: No bootstrap required, opened a new log
I20260810 03:26:48.403107 8088 ts_tablet_manager.cc:1397] T 0df834d8e3484e3fa617e27fa69f808a P db6b5c27fe7b480f96dbf4e5afd7b6b9: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260810 03:26:48.403553 8089 tablet_bootstrap.cc:654] T 0df834d8e3484e3fa617e27fa69f808a P 9a1b431e2e804faab2004da408cd20f8: Neither blocks nor log segments found. Creating new log.
I20260810 03:26:48.404091 8088 raft_consensus.cc:348] T 0df834d8e3484e3fa617e27fa69f808a P db6b5c27fe7b480f96dbf4e5afd7b6b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51a5b515913a4b10bd40695507e0fcca" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 35325 } } peers { permanent_uuid: "db6b5c27fe7b480f96dbf4e5afd7b6b9" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 33609 } } peers { permanent_uuid: "9a1b431e2e804faab2004da408cd20f8" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 40179 } }
I20260810 03:26:48.404418 8088 raft_consensus.cc:374] T 0df834d8e3484e3fa617e27fa69f808a P db6b5c27fe7b480f96dbf4e5afd7b6b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:26:48.404502 8088 raft_consensus.cc:729] T 0df834d8e3484e3fa617e27fa69f808a P db6b5c27fe7b480f96dbf4e5afd7b6b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db6b5c27fe7b480f96dbf4e5afd7b6b9, State: Initialized, Role: FOLLOWER
I20260810 03:26:48.404795 8088 consensus_queue.cc:260] T 0df834d8e3484e3fa617e27fa69f808a P db6b5c27fe7b480f96dbf4e5afd7b6b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51a5b515913a4b10bd40695507e0fcca" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 35325 } } peers { permanent_uuid: "db6b5c27fe7b480f96dbf4e5afd7b6b9" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 33609 } } peers { permanent_uuid: "9a1b431e2e804faab2004da408cd20f8" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 40179 } }
I20260810 03:26:48.405433 8098 tablet_replica.cc:439] T 0df834d8e3484e3fa617e27fa69f808a P db6b5c27fe7b480f96dbf4e5afd7b6b9: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51a5b515913a4b10bd40695507e0fcca" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 35325 } } peers { permanent_uuid: "db6b5c27fe7b480f96dbf4e5afd7b6b9" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 33609 } } peers { permanent_uuid: "9a1b431e2e804faab2004da408cd20f8" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 40179 } } }
I20260810 03:26:48.405735 8098 tablet_replica.cc:442] T 0df834d8e3484e3fa617e27fa69f808a P db6b5c27fe7b480f96dbf4e5afd7b6b9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 03:26:48.409209 8088 ts_tablet_manager.cc:1428] T 0df834d8e3484e3fa617e27fa69f808a P db6b5c27fe7b480f96dbf4e5afd7b6b9: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20260810 03:26:48.414597 8089 tablet_bootstrap.cc:492] T 0df834d8e3484e3fa617e27fa69f808a P 9a1b431e2e804faab2004da408cd20f8: No bootstrap required, opened a new log
I20260810 03:26:48.414755 8092 tablet_bootstrap.cc:492] T 0df834d8e3484e3fa617e27fa69f808a P 51a5b515913a4b10bd40695507e0fcca: No bootstrap required, opened a new log
I20260810 03:26:48.414817 8089 ts_tablet_manager.cc:1397] T 0df834d8e3484e3fa617e27fa69f808a P 9a1b431e2e804faab2004da408cd20f8: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.003s
I20260810 03:26:48.414969 8092 ts_tablet_manager.cc:1397] T 0df834d8e3484e3fa617e27fa69f808a P 51a5b515913a4b10bd40695507e0fcca: Time spent bootstrapping tablet: real 0.018s user 0.004s sys 0.011s
I20260810 03:26:48.415518 8089 raft_consensus.cc:348] T 0df834d8e3484e3fa617e27fa69f808a P 9a1b431e2e804faab2004da408cd20f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51a5b515913a4b10bd40695507e0fcca" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 35325 } } peers { permanent_uuid: "db6b5c27fe7b480f96dbf4e5afd7b6b9" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 33609 } } peers { permanent_uuid: "9a1b431e2e804faab2004da408cd20f8" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 40179 } }
I20260810 03:26:48.415755 8089 raft_consensus.cc:374] T 0df834d8e3484e3fa617e27fa69f808a P 9a1b431e2e804faab2004da408cd20f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:26:48.415839 8089 raft_consensus.cc:729] T 0df834d8e3484e3fa617e27fa69f808a P 9a1b431e2e804faab2004da408cd20f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a1b431e2e804faab2004da408cd20f8, State: Initialized, Role: FOLLOWER
I20260810 03:26:48.415771 8092 raft_consensus.cc:348] T 0df834d8e3484e3fa617e27fa69f808a P 51a5b515913a4b10bd40695507e0fcca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51a5b515913a4b10bd40695507e0fcca" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 35325 } } peers { permanent_uuid: "db6b5c27fe7b480f96dbf4e5afd7b6b9" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 33609 } } peers { permanent_uuid: "9a1b431e2e804faab2004da408cd20f8" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 40179 } }
I20260810 03:26:48.416010 8092 raft_consensus.cc:374] T 0df834d8e3484e3fa617e27fa69f808a P 51a5b515913a4b10bd40695507e0fcca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:26:48.416116 8092 raft_consensus.cc:729] T 0df834d8e3484e3fa617e27fa69f808a P 51a5b515913a4b10bd40695507e0fcca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51a5b515913a4b10bd40695507e0fcca, State: Initialized, Role: FOLLOWER
I20260810 03:26:48.416325 8089 consensus_queue.cc:260] T 0df834d8e3484e3fa617e27fa69f808a P 9a1b431e2e804faab2004da408cd20f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51a5b515913a4b10bd40695507e0fcca" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 35325 } } peers { permanent_uuid: "db6b5c27fe7b480f96dbf4e5afd7b6b9" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 33609 } } peers { permanent_uuid: "9a1b431e2e804faab2004da408cd20f8" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 40179 } }
I20260810 03:26:48.416468 8092 consensus_queue.cc:260] T 0df834d8e3484e3fa617e27fa69f808a P 51a5b515913a4b10bd40695507e0fcca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51a5b515913a4b10bd40695507e0fcca" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 35325 } } peers { permanent_uuid: "db6b5c27fe7b480f96dbf4e5afd7b6b9" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 33609 } } peers { permanent_uuid: "9a1b431e2e804faab2004da408cd20f8" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 40179 } }
I20260810 03:26:48.416865 8097 tablet_replica.cc:439] T 0df834d8e3484e3fa617e27fa69f808a P 9a1b431e2e804faab2004da408cd20f8: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51a5b515913a4b10bd40695507e0fcca" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 35325 } } peers { permanent_uuid: "db6b5c27fe7b480f96dbf4e5afd7b6b9" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 33609 } } peers { permanent_uuid: "9a1b431e2e804faab2004da408cd20f8" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 40179 } } }
I20260810 03:26:48.417069 8092 ts_tablet_manager.cc:1428] T 0df834d8e3484e3fa617e27fa69f808a P 51a5b515913a4b10bd40695507e0fcca: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260810 03:26:48.417137 8097 tablet_replica.cc:442] T 0df834d8e3484e3fa617e27fa69f808a P 9a1b431e2e804faab2004da408cd20f8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 03:26:48.417511 8099 tablet_replica.cc:439] T 0df834d8e3484e3fa617e27fa69f808a P 51a5b515913a4b10bd40695507e0fcca: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51a5b515913a4b10bd40695507e0fcca" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 35325 } } peers { permanent_uuid: "db6b5c27fe7b480f96dbf4e5afd7b6b9" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 33609 } } peers { permanent_uuid: "9a1b431e2e804faab2004da408cd20f8" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 40179 } } }
I20260810 03:26:48.417887 8099 tablet_replica.cc:442] T 0df834d8e3484e3fa617e27fa69f808a P 51a5b515913a4b10bd40695507e0fcca: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 03:26:48.419044 8089 ts_tablet_manager.cc:1428] T 0df834d8e3484e3fa617e27fa69f808a P 9a1b431e2e804faab2004da408cd20f8: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20260810 03:26:48.421936 8099 raft_consensus.cc:482] T 0df834d8e3484e3fa617e27fa69f808a P 51a5b515913a4b10bd40695507e0fcca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 03:26:48.422174 8099 raft_consensus.cc:504] T 0df834d8e3484e3fa617e27fa69f808a P 51a5b515913a4b10bd40695507e0fcca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51a5b515913a4b10bd40695507e0fcca" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 35325 } } peers { permanent_uuid: "db6b5c27fe7b480f96dbf4e5afd7b6b9" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 33609 } } peers { permanent_uuid: "9a1b431e2e804faab2004da408cd20f8" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 40179 } }
I20260810 03:26:48.424118 8099 leader_election.cc:288] T 0df834d8e3484e3fa617e27fa69f808a P 51a5b515913a4b10bd40695507e0fcca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db6b5c27fe7b480f96dbf4e5afd7b6b9 (127.27.161.2:33609), 9a1b431e2e804faab2004da408cd20f8 (127.27.161.3:40179)
W20260810 03:26:48.428610 7645 tablet.cc:2369] T 0df834d8e3484e3fa617e27fa69f808a P 51a5b515913a4b10bd40695507e0fcca: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 03:26:48.433049 8145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:48.424314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.2:33609 (local address 127.27.161.1:43235)
0810 03:26:48.425405 (+ 1091us) negotiation.cc:107] Waiting for socket to connect
0810 03:26:48.425427 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0810 03:26:48.425651 (+ 224us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:26:48.426296 (+ 645us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:26:48.426302 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:26:48.426330 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:26:48.426645 (+ 315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:26:48.426674 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:26:48.428176 (+ 1502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:26:48.428182 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:26:48.430319 (+ 2137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:26:48.430335 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:26:48.432199 (+ 1864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:26:48.432206 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:26:48.432397 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:48.432434 (+ 37us) client_negotiation.cc:769] Sending connection context
0810 03:26:48.432539 (+ 105us) client_negotiation.cc:240] Negotiation successful
0810 03:26:48.432689 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":919,"thread_start_us":629,"threads_started":1}
I20260810 03:26:48.433038 8083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:48.424615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.161.1:43235 (local address 127.27.161.2:33609)
0810 03:26:48.425189 (+ 574us) server_negotiation.cc:206] Beginning negotiation
0810 03:26:48.425196 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:26:48.425518 (+ 322us) server_negotiation.cc:407] Connection header received
0810 03:26:48.425837 (+ 319us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:26:48.425843 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:26:48.425991 (+ 148us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:26:48.426074 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:26:48.426870 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:48.427965 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:48.430522 (+ 2557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:48.431898 (+ 1376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:48.432101 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:48.432216 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0810 03:26:48.432594 (+ 378us) server_negotiation.cc:299] Negotiation successful
0810 03:26:48.432749 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":382}
I20260810 03:26:48.434126 7779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0df834d8e3484e3fa617e27fa69f808a" candidate_uuid: "51a5b515913a4b10bd40695507e0fcca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db6b5c27fe7b480f96dbf4e5afd7b6b9" is_pre_election: true
I20260810 03:26:48.434469 7779 raft_consensus.cc:2454] T 0df834d8e3484e3fa617e27fa69f808a P db6b5c27fe7b480f96dbf4e5afd7b6b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51a5b515913a4b10bd40695507e0fcca in term 0.
I20260810 03:26:48.435060 7460 leader_election.cc:302] T 0df834d8e3484e3fa617e27fa69f808a P 51a5b515913a4b10bd40695507e0fcca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51a5b515913a4b10bd40695507e0fcca, db6b5c27fe7b480f96dbf4e5afd7b6b9; no voters:
I20260810 03:26:48.435665 8099 raft_consensus.cc:2781] T 0df834d8e3484e3fa617e27fa69f808a P 51a5b515913a4b10bd40695507e0fcca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 03:26:48.435803 8099 raft_consensus.cc:482] T 0df834d8e3484e3fa617e27fa69f808a P 51a5b515913a4b10bd40695507e0fcca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 03:26:48.435905 8099 raft_consensus.cc:3037] T 0df834d8e3484e3fa617e27fa69f808a P 51a5b515913a4b10bd40695507e0fcca [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:26:48.437834 8085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:48.426074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.161.1:56671 (local address 127.27.161.3:40179)
0810 03:26:48.426253 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0810 03:26:48.426259 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:26:48.431064 (+ 4805us) server_negotiation.cc:407] Connection header received
0810 03:26:48.431248 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:26:48.431254 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:26:48.431337 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:26:48.431439 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:26:48.432148 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:48.433159 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:48.435353 (+ 2194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:48.436549 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:48.436699 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:48.436774 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0810 03:26:48.437409 (+ 635us) server_negotiation.cc:299] Negotiation successful
0810 03:26:48.437558 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260810 03:26:48.437886 8146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:48.425514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.3:40179 (local address 127.27.161.1:56671)
0810 03:26:48.430942 (+ 5428us) negotiation.cc:107] Waiting for socket to connect
0810 03:26:48.430963 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0810 03:26:48.431116 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:26:48.431468 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:26:48.431474 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:26:48.431503 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:26:48.431916 (+ 413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:26:48.431929 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:26:48.433347 (+ 1418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:26:48.433352 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:26:48.435161 (+ 1809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:26:48.435178 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:26:48.436998 (+ 1820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:26:48.437004 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:26:48.437206 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:48.437244 (+ 38us) client_negotiation.cc:769] Sending connection context
0810 03:26:48.437346 (+ 102us) client_negotiation.cc:240] Negotiation successful
0810 03:26:48.437500 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5290,"thread_start_us":5062,"threads_started":1}
I20260810 03:26:48.438671 7977 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0df834d8e3484e3fa617e27fa69f808a" candidate_uuid: "51a5b515913a4b10bd40695507e0fcca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a1b431e2e804faab2004da408cd20f8" is_pre_election: true
I20260810 03:26:48.438933 7977 raft_consensus.cc:2454] T 0df834d8e3484e3fa617e27fa69f808a P 9a1b431e2e804faab2004da408cd20f8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51a5b515913a4b10bd40695507e0fcca in term 0.
I20260810 03:26:48.439244 8099 raft_consensus.cc:504] T 0df834d8e3484e3fa617e27fa69f808a P 51a5b515913a4b10bd40695507e0fcca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51a5b515913a4b10bd40695507e0fcca" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 35325 } } peers { permanent_uuid: "db6b5c27fe7b480f96dbf4e5afd7b6b9" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 33609 } } peers { permanent_uuid: "9a1b431e2e804faab2004da408cd20f8" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 40179 } }
I20260810 03:26:48.439965 8099 leader_election.cc:288] T 0df834d8e3484e3fa617e27fa69f808a P 51a5b515913a4b10bd40695507e0fcca [CANDIDATE]: Term 1 election: Requested vote from peers db6b5c27fe7b480f96dbf4e5afd7b6b9 (127.27.161.2:33609), 9a1b431e2e804faab2004da408cd20f8 (127.27.161.3:40179)
I20260810 03:26:48.440492 7977 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0df834d8e3484e3fa617e27fa69f808a" candidate_uuid: "51a5b515913a4b10bd40695507e0fcca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a1b431e2e804faab2004da408cd20f8"
I20260810 03:26:48.440546 7779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0df834d8e3484e3fa617e27fa69f808a" candidate_uuid: "51a5b515913a4b10bd40695507e0fcca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db6b5c27fe7b480f96dbf4e5afd7b6b9"
I20260810 03:26:48.440714 7977 raft_consensus.cc:3037] T 0df834d8e3484e3fa617e27fa69f808a P 9a1b431e2e804faab2004da408cd20f8 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:26:48.440768 7779 raft_consensus.cc:3037] T 0df834d8e3484e3fa617e27fa69f808a P db6b5c27fe7b480f96dbf4e5afd7b6b9 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:26:48.443905 7977 raft_consensus.cc:2454] T 0df834d8e3484e3fa617e27fa69f808a P 9a1b431e2e804faab2004da408cd20f8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51a5b515913a4b10bd40695507e0fcca in term 1.
I20260810 03:26:48.444237 7779 raft_consensus.cc:2454] T 0df834d8e3484e3fa617e27fa69f808a P db6b5c27fe7b480f96dbf4e5afd7b6b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51a5b515913a4b10bd40695507e0fcca in term 1.
I20260810 03:26:48.444381 7460 leader_election.cc:302] T 0df834d8e3484e3fa617e27fa69f808a P 51a5b515913a4b10bd40695507e0fcca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51a5b515913a4b10bd40695507e0fcca, 9a1b431e2e804faab2004da408cd20f8; no voters:
I20260810 03:26:48.445029 8099 raft_consensus.cc:2781] T 0df834d8e3484e3fa617e27fa69f808a P 51a5b515913a4b10bd40695507e0fcca [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:26:48.445626 8099 raft_consensus.cc:686] T 0df834d8e3484e3fa617e27fa69f808a P 51a5b515913a4b10bd40695507e0fcca [term 1 LEADER]: Becoming Leader. State: Replica: 51a5b515913a4b10bd40695507e0fcca, State: Running, Role: LEADER
I20260810 03:26:48.446121 8099 consensus_queue.cc:237] T 0df834d8e3484e3fa617e27fa69f808a P 51a5b515913a4b10bd40695507e0fcca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51a5b515913a4b10bd40695507e0fcca" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 35325 } } peers { permanent_uuid: "db6b5c27fe7b480f96dbf4e5afd7b6b9" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 33609 } } peers { permanent_uuid: "9a1b431e2e804faab2004da408cd20f8" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 40179 } }
I20260810 03:26:48.448098 8148 tablet_replica.cc:439] T 0df834d8e3484e3fa617e27fa69f808a P 51a5b515913a4b10bd40695507e0fcca: TxnStatusTablet state changed. Reason: New leader 51a5b515913a4b10bd40695507e0fcca. Latest consensus state: current_term: 1 leader_uuid: "51a5b515913a4b10bd40695507e0fcca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51a5b515913a4b10bd40695507e0fcca" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 35325 } } peers { permanent_uuid: "db6b5c27fe7b480f96dbf4e5afd7b6b9" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 33609 } } peers { permanent_uuid: "9a1b431e2e804faab2004da408cd20f8" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 40179 } } }
I20260810 03:26:48.448572 8148 tablet_replica.cc:442] T 0df834d8e3484e3fa617e27fa69f808a P 51a5b515913a4b10bd40695507e0fcca: This TxnStatusTablet replica's current role is: LEADER
I20260810 03:26:48.449384 8150 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260810 03:26:48.449589 8150 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260810 03:26:48.450055 7318 catalog_manager.cc:5495] T 0df834d8e3484e3fa617e27fa69f808a P 51a5b515913a4b10bd40695507e0fcca reported cstate change: term changed from 0 to 1, leader changed from <none> to 51a5b515913a4b10bd40695507e0fcca (127.27.161.1). New cstate: current_term: 1 leader_uuid: "51a5b515913a4b10bd40695507e0fcca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51a5b515913a4b10bd40695507e0fcca" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 35325 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "db6b5c27fe7b480f96dbf4e5afd7b6b9" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 33609 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9a1b431e2e804faab2004da408cd20f8" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 40179 } health_report { overall_health: UNKNOWN } } }
I20260810 03:26:48.475972 8087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:48.468252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38532 (local address 127.27.161.1:35325)
0810 03:26:48.468461 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0810 03:26:48.468467 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:26:48.468488 (+ 21us) server_negotiation.cc:407] Connection header received
0810 03:26:48.468592 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:26:48.468597 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:26:48.468699 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:26:48.468846 (+ 147us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 03:26:48.469652 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:48.470932 (+ 1280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:48.472896 (+ 1964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:48.473571 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:48.473707 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:48.474358 (+ 651us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 03:26:48.475217 (+ 859us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 03:26:48.475368 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0810 03:26:48.475528 (+ 160us) server_negotiation.cc:299] Negotiation successful
0810 03:26:48.475640 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260810 03:26:48.476002 8141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:48.467837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.1:35325 (local address 127.0.0.1:38532)
0810 03:26:48.468147 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0810 03:26:48.468170 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0810 03:26:48.468323 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:26:48.469238 (+ 915us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:26:48.469243 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:26:48.469257 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 03:26:48.469463 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:26:48.469476 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:26:48.471118 (+ 1642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:26:48.471125 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:26:48.472700 (+ 1575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:26:48.472720 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:26:48.473778 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:26:48.473785 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:26:48.474022 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:48.474127 (+ 105us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 03:26:48.475353 (+ 1226us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 03:26:48.475388 (+ 35us) client_negotiation.cc:769] Sending connection context
0810 03:26:48.475491 (+ 103us) client_negotiation.cc:240] Negotiation successful
0810 03:26:48.475691 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":68}
I20260810 03:26:48.481165 7977 raft_consensus.cc:1261] T 0df834d8e3484e3fa617e27fa69f808a P 9a1b431e2e804faab2004da408cd20f8 [term 1 FOLLOWER]: Refusing update from remote peer 51a5b515913a4b10bd40695507e0fcca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:26:48.481587 7779 raft_consensus.cc:1261] T 0df834d8e3484e3fa617e27fa69f808a P db6b5c27fe7b480f96dbf4e5afd7b6b9 [term 1 FOLLOWER]: Refusing update from remote peer 51a5b515913a4b10bd40695507e0fcca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:26:48.482372 8148 consensus_queue.cc:1035] T 0df834d8e3484e3fa617e27fa69f808a P 51a5b515913a4b10bd40695507e0fcca [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a1b431e2e804faab2004da408cd20f8" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 40179 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:26:48.482807 8099 consensus_queue.cc:1035] T 0df834d8e3484e3fa617e27fa69f808a P 51a5b515913a4b10bd40695507e0fcca [LEADER]: Connected to new peer: Peer: permanent_uuid: "db6b5c27fe7b480f96dbf4e5afd7b6b9" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 33609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:26:48.496744 8097 tablet_replica.cc:439] T 0df834d8e3484e3fa617e27fa69f808a P 9a1b431e2e804faab2004da408cd20f8: TxnStatusTablet state changed. Reason: New leader 51a5b515913a4b10bd40695507e0fcca. Latest consensus state: current_term: 1 leader_uuid: "51a5b515913a4b10bd40695507e0fcca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51a5b515913a4b10bd40695507e0fcca" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 35325 } } peers { permanent_uuid: "db6b5c27fe7b480f96dbf4e5afd7b6b9" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 33609 } } peers { permanent_uuid: "9a1b431e2e804faab2004da408cd20f8" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 40179 } } }
I20260810 03:26:48.497115 8097 tablet_replica.cc:442] T 0df834d8e3484e3fa617e27fa69f808a P 9a1b431e2e804faab2004da408cd20f8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 03:26:48.498983 8098 tablet_replica.cc:439] T 0df834d8e3484e3fa617e27fa69f808a P db6b5c27fe7b480f96dbf4e5afd7b6b9: TxnStatusTablet state changed. Reason: New leader 51a5b515913a4b10bd40695507e0fcca. Latest consensus state: current_term: 1 leader_uuid: "51a5b515913a4b10bd40695507e0fcca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51a5b515913a4b10bd40695507e0fcca" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 35325 } } peers { permanent_uuid: "db6b5c27fe7b480f96dbf4e5afd7b6b9" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 33609 } } peers { permanent_uuid: "9a1b431e2e804faab2004da408cd20f8" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 40179 } } }
I20260810 03:26:48.499328 8098 tablet_replica.cc:442] T 0df834d8e3484e3fa617e27fa69f808a P db6b5c27fe7b480f96dbf4e5afd7b6b9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 03:26:48.501142 8112 mvcc.cc:205] Tried to move back new op lower bound from 7316817545131012096 to 7316817544998912000. Current Snapshot: MvccSnapshot[applied={T|T < 7316817545131012096}]
I20260810 03:26:48.516861 8099 tablet_replica.cc:439] T 0df834d8e3484e3fa617e27fa69f808a P 51a5b515913a4b10bd40695507e0fcca: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "51a5b515913a4b10bd40695507e0fcca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51a5b515913a4b10bd40695507e0fcca" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 35325 } } peers { permanent_uuid: "db6b5c27fe7b480f96dbf4e5afd7b6b9" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 33609 } } peers { permanent_uuid: "9a1b431e2e804faab2004da408cd20f8" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 40179 } } }
I20260810 03:26:48.518122 8099 tablet_replica.cc:442] T 0df834d8e3484e3fa617e27fa69f808a P 51a5b515913a4b10bd40695507e0fcca: This TxnStatusTablet replica's current role is: LEADER
I20260810 03:26:48.518904 8097 tablet_replica.cc:439] T 0df834d8e3484e3fa617e27fa69f808a P 9a1b431e2e804faab2004da408cd20f8: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "51a5b515913a4b10bd40695507e0fcca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51a5b515913a4b10bd40695507e0fcca" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 35325 } } peers { permanent_uuid: "db6b5c27fe7b480f96dbf4e5afd7b6b9" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 33609 } } peers { permanent_uuid: "9a1b431e2e804faab2004da408cd20f8" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 40179 } } }
I20260810 03:26:48.519218 8097 tablet_replica.cc:442] T 0df834d8e3484e3fa617e27fa69f808a P 9a1b431e2e804faab2004da408cd20f8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 03:26:48.519577 8085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:48.509642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51204 (local address 127.27.161.3:40179)
0810 03:26:48.509862 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0810 03:26:48.509867 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:26:48.509887 (+ 20us) server_negotiation.cc:407] Connection header received
0810 03:26:48.510045 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:26:48.510050 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:26:48.510161 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:26:48.510324 (+ 163us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 03:26:48.510979 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:48.512086 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:48.514229 (+ 2143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:48.514903 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:48.515053 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:48.516239 (+ 1186us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 03:26:48.518247 (+ 2008us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 03:26:48.518414 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0810 03:26:48.519143 (+ 729us) server_negotiation.cc:299] Negotiation successful
0810 03:26:48.519246 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260810 03:26:48.522256 8111 mvcc.cc:205] Tried to move back new op lower bound from 7316817545131012096 to 7316817544998912000. Current Snapshot: MvccSnapshot[applied={T|T < 7316817545131012096 or (T in {7316817545131012096})}]
I20260810 03:26:48.523382 8098 tablet_replica.cc:439] T 0df834d8e3484e3fa617e27fa69f808a P db6b5c27fe7b480f96dbf4e5afd7b6b9: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "51a5b515913a4b10bd40695507e0fcca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51a5b515913a4b10bd40695507e0fcca" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 35325 } } peers { permanent_uuid: "db6b5c27fe7b480f96dbf4e5afd7b6b9" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 33609 } } peers { permanent_uuid: "9a1b431e2e804faab2004da408cd20f8" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 40179 } } }
I20260810 03:26:48.523669 8098 tablet_replica.cc:442] T 0df834d8e3484e3fa617e27fa69f808a P db6b5c27fe7b480f96dbf4e5afd7b6b9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 03:26:48.523653 8162 tablet_replica.cc:439] T 0df834d8e3484e3fa617e27fa69f808a P 51a5b515913a4b10bd40695507e0fcca: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "51a5b515913a4b10bd40695507e0fcca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51a5b515913a4b10bd40695507e0fcca" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 35325 } } peers { permanent_uuid: "db6b5c27fe7b480f96dbf4e5afd7b6b9" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 33609 } } peers { permanent_uuid: "9a1b431e2e804faab2004da408cd20f8" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 40179 } } }
I20260810 03:26:48.524020 8162 tablet_replica.cc:442] T 0df834d8e3484e3fa617e27fa69f808a P 51a5b515913a4b10bd40695507e0fcca: This TxnStatusTablet replica's current role is: LEADER
I20260810 03:26:48.550501 8087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:48.544541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.161.3:57667 (local address 127.27.161.1:35325)
0810 03:26:48.544737 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0810 03:26:48.544743 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:26:48.545332 (+ 589us) server_negotiation.cc:407] Connection header received
0810 03:26:48.545539 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:26:48.545543 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:26:48.545612 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:26:48.545695 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 03:26:48.546257 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:48.546993 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:48.548501 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:48.549072 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:48.549194 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:48.549677 (+ 483us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 03:26:48.549811 (+ 134us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 03:26:48.549885 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0810 03:26:48.550088 (+ 203us) server_negotiation.cc:299] Negotiation successful
0810 03:26:48.550175 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260810 03:26:48.550673 8167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:48.544537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.1:35325 (local address 127.27.161.3:57667)
0810 03:26:48.545224 (+ 687us) negotiation.cc:107] Waiting for socket to connect
0810 03:26:48.545240 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 03:26:48.545400 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:26:48.545726 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:26:48.545731 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:26:48.545744 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 03:26:48.546091 (+ 347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:26:48.546102 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:26:48.547122 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:26:48.547128 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:26:48.548309 (+ 1181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:26:48.548324 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:26:48.549249 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:26:48.549256 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:26:48.549394 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:48.549461 (+ 67us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 03:26:48.549907 (+ 446us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 03:26:48.549919 (+ 12us) client_negotiation.cc:769] Sending connection context
0810 03:26:48.550060 (+ 141us) client_negotiation.cc:240] Negotiation successful
0810 03:26:48.550260 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":520,"thread_start_us":366,"threads_started":1}
I20260810 03:26:48.562409 7779 raft_consensus.cc:1261] T a281469115a94ba281a7bf9772255ebc P db6b5c27fe7b480f96dbf4e5afd7b6b9 [term 1 FOLLOWER]: Refusing update from remote peer 9a1b431e2e804faab2004da408cd20f8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:26:48.562633 7570 raft_consensus.cc:1261] T a281469115a94ba281a7bf9772255ebc P 51a5b515913a4b10bd40695507e0fcca [term 1 FOLLOWER]: Refusing update from remote peer 9a1b431e2e804faab2004da408cd20f8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:26:48.563354 8131 consensus_queue.cc:1035] T a281469115a94ba281a7bf9772255ebc P 9a1b431e2e804faab2004da408cd20f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51a5b515913a4b10bd40695507e0fcca" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 35325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:26:48.563764 8097 consensus_queue.cc:1035] T a281469115a94ba281a7bf9772255ebc P 9a1b431e2e804faab2004da408cd20f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db6b5c27fe7b480f96dbf4e5afd7b6b9" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 33609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:26:48.635322 8085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:48.628397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.161.1:50161 (local address 127.27.161.3:40179)
0810 03:26:48.628582 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0810 03:26:48.628588 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:26:48.628815 (+ 227us) server_negotiation.cc:407] Connection header received
0810 03:26:48.629195 (+ 380us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:26:48.629201 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:26:48.629293 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:26:48.629356 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 03:26:48.630207 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:48.630997 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:48.633269 (+ 2272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:48.633987 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:48.634125 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:48.634567 (+ 442us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 03:26:48.634696 (+ 129us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 03:26:48.634770 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0810 03:26:48.634912 (+ 142us) server_negotiation.cc:299] Negotiation successful
0810 03:26:48.634999 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260810 03:26:48.635474 8174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:48.627891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.3:40179 (local address 127.27.161.1:50161)
0810 03:26:48.628723 (+ 832us) negotiation.cc:107] Waiting for socket to connect
0810 03:26:48.628741 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 03:26:48.628944 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:26:48.629555 (+ 611us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:26:48.629560 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:26:48.629573 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 03:26:48.630026 (+ 453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:26:48.630040 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:26:48.631145 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:26:48.631151 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:26:48.633099 (+ 1948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:26:48.633115 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:26:48.634174 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:26:48.634182 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:26:48.634313 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:48.634379 (+ 66us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 03:26:48.634795 (+ 416us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 03:26:48.634807 (+ 12us) client_negotiation.cc:769] Sending connection context
0810 03:26:48.634884 (+ 77us) client_negotiation.cc:240] Negotiation successful
0810 03:26:48.635042 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":576,"thread_start_us":1615,"threads_started":1}
I20260810 03:26:48.703940 7320 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33372:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 03:26:48.704258 7320 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33372:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 03:26:48.708745 7320 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6882776d888e4399b85872d4661cecf3: Sending DeleteTablet for 3 replicas of tablet a281469115a94ba281a7bf9772255ebc
I20260810 03:26:48.710289 7542 tablet_service.cc:1514] Processing DeleteTablet for tablet a281469115a94ba281a7bf9772255ebc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:26:48 UTC) from {username='oryx', principal='oryx/127.27.161.62@KRBTEST.COM'} at 127.0.0.1:38524
I20260810 03:26:48.711683 7742 tablet_service.cc:1514] Processing DeleteTablet for tablet a281469115a94ba281a7bf9772255ebc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:26:48 UTC) from {username='oryx', principal='oryx/127.27.161.62@KRBTEST.COM'} at 127.0.0.1:36816
I20260810 03:26:48.712477 7946 tablet_service.cc:1514] Processing DeleteTablet for tablet a281469115a94ba281a7bf9772255ebc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:26:48 UTC) from {username='oryx', principal='oryx/127.27.161.62@KRBTEST.COM'} at 127.0.0.1:51190
I20260810 03:26:48.727385 8183 tablet_replica.cc:330] T a281469115a94ba281a7bf9772255ebc P db6b5c27fe7b480f96dbf4e5afd7b6b9: stopping tablet replica
I20260810 03:26:48.727664 8183 raft_consensus.cc:2229] T a281469115a94ba281a7bf9772255ebc P db6b5c27fe7b480f96dbf4e5afd7b6b9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 03:26:48.727926 8183 raft_consensus.cc:2258] T a281469115a94ba281a7bf9772255ebc P db6b5c27fe7b480f96dbf4e5afd7b6b9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:26:48.729344 8183 ts_tablet_manager.cc:1905] T a281469115a94ba281a7bf9772255ebc P db6b5c27fe7b480f96dbf4e5afd7b6b9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:26:48.737592 8183 ts_tablet_manager.cc:1918] T a281469115a94ba281a7bf9772255ebc P db6b5c27fe7b480f96dbf4e5afd7b6b9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 03:26:48.749538 8183 log.cc:1196] T a281469115a94ba281a7bf9772255ebc P db6b5c27fe7b480f96dbf4e5afd7b6b9: Deleting WAL directory at /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786332336319544-28292-0/minicluster-data/ts-1/wal/wals/a281469115a94ba281a7bf9772255ebc
I20260810 03:26:48.750340 8183 ts_tablet_manager.cc:1939] T a281469115a94ba281a7bf9772255ebc P db6b5c27fe7b480f96dbf4e5afd7b6b9: Deleting consensus metadata
I20260810 03:26:48.751945 7293 catalog_manager.cc:4841] TS db6b5c27fe7b480f96dbf4e5afd7b6b9 (127.27.161.2:33609): tablet a281469115a94ba281a7bf9772255ebc (table test-table [id=c892b1cc7de74c9a862af6065f116311]) successfully deleted
I20260810 03:26:48.755470 8182 tablet_replica.cc:330] T a281469115a94ba281a7bf9772255ebc P 51a5b515913a4b10bd40695507e0fcca: stopping tablet replica
I20260810 03:26:48.755767 8185 tablet_replica.cc:330] T a281469115a94ba281a7bf9772255ebc P 9a1b431e2e804faab2004da408cd20f8: stopping tablet replica
I20260810 03:26:48.756039 8185 raft_consensus.cc:2229] T a281469115a94ba281a7bf9772255ebc P 9a1b431e2e804faab2004da408cd20f8 [term 1 LEADER]: Raft consensus shutting down.
I20260810 03:26:48.756958 8182 raft_consensus.cc:2229] T a281469115a94ba281a7bf9772255ebc P 51a5b515913a4b10bd40695507e0fcca [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 03:26:48.757294 8182 raft_consensus.cc:2258] T a281469115a94ba281a7bf9772255ebc P 51a5b515913a4b10bd40695507e0fcca [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:26:48.757894 8185 raft_consensus.cc:2258] T a281469115a94ba281a7bf9772255ebc P 9a1b431e2e804faab2004da408cd20f8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:26:48.759363 8185 ts_tablet_manager.cc:1905] T a281469115a94ba281a7bf9772255ebc P 9a1b431e2e804faab2004da408cd20f8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:26:48.763413 8182 ts_tablet_manager.cc:1905] T a281469115a94ba281a7bf9772255ebc P 51a5b515913a4b10bd40695507e0fcca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:26:48.770047 8185 ts_tablet_manager.cc:1918] T a281469115a94ba281a7bf9772255ebc P 9a1b431e2e804faab2004da408cd20f8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 03:26:48.771723 8182 ts_tablet_manager.cc:1918] T a281469115a94ba281a7bf9772255ebc P 51a5b515913a4b10bd40695507e0fcca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 03:26:48.771889 8182 log.cc:1196] T a281469115a94ba281a7bf9772255ebc P 51a5b515913a4b10bd40695507e0fcca: Deleting WAL directory at /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786332336319544-28292-0/minicluster-data/ts-0/wal/wals/a281469115a94ba281a7bf9772255ebc
I20260810 03:26:48.772426 8182 ts_tablet_manager.cc:1939] T a281469115a94ba281a7bf9772255ebc P 51a5b515913a4b10bd40695507e0fcca: Deleting consensus metadata
I20260810 03:26:48.773792 7294 catalog_manager.cc:4841] TS 51a5b515913a4b10bd40695507e0fcca (127.27.161.1:35325): tablet a281469115a94ba281a7bf9772255ebc (table test-table [id=c892b1cc7de74c9a862af6065f116311]) successfully deleted
I20260810 03:26:48.774322 8185 log.cc:1196] T a281469115a94ba281a7bf9772255ebc P 9a1b431e2e804faab2004da408cd20f8: Deleting WAL directory at /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786332336319544-28292-0/minicluster-data/ts-2/wal/wals/a281469115a94ba281a7bf9772255ebc
I20260810 03:26:48.774849 8185 ts_tablet_manager.cc:1939] T a281469115a94ba281a7bf9772255ebc P 9a1b431e2e804faab2004da408cd20f8: Deleting consensus metadata
I20260810 03:26:48.776170 7293 catalog_manager.cc:4841] TS 9a1b431e2e804faab2004da408cd20f8 (127.27.161.3:40179): tablet a281469115a94ba281a7bf9772255ebc (table test-table [id=c892b1cc7de74c9a862af6065f116311]) successfully deleted
Aug 10 03:26:48 dist-test-slave-vpvm krb5kdc[7252](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332408, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
W20260810 03:26:48.831494 28292 mini_kdc.cc:317] Time spent kinit for test-admin: real 0.109s user 0.000s sys 0.026s
Aug 10 03:26:48 dist-test-slave-vpvm krb5kdc[7252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.27.161.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 03:26:48 dist-test-slave-vpvm krb5kdc[7252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.27.161.62@KRBTEST.COM, Server not found in Kerberos database
W20260810 03:26:48.886843 7891 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:26:48.871351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53860 (local address 127.27.161.62:35965)
0810 03:26:48.871555 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0810 03:26:48.871561 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:26:48.872160 (+ 599us) server_negotiation.cc:407] Connection header received
0810 03:26:48.872374 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:26:48.872380 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:26:48.872486 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:26:48.872595 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:26:48.874567 (+ 1972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:48.875790 (+ 1223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:48.877945 (+ 2155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:48.878685 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:48.878836 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:48.886568 (+ 7732us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53860: BlockingRecv error: recv got EOF from 127.0.0.1:53860 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Aug 10 03:26:48 dist-test-slave-vpvm krb5kdc[7252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332408, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.27.161.62@KRBTEST.COM
I20260810 03:26:48.911010 7891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:48.891246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53870 (local address 127.27.161.62:35965)
0810 03:26:48.891434 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0810 03:26:48.891440 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:26:48.891559 (+ 119us) server_negotiation.cc:407] Connection header received
0810 03:26:48.891872 (+ 313us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:26:48.891878 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:26:48.891979 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:26:48.892090 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:26:48.893854 (+ 1764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:48.895024 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:48.896924 (+ 1900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:48.897601 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:48.897740 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:48.903631 (+ 5891us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:26:48.903640 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:26:48.903653 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:26:48.903712 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:26:48.907516 (+ 3804us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:26:48.908486 (+ 970us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:26:48.908493 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:26:48.908497 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:26:48.908703 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:26:48.909243 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:26:48.909248 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:26:48.909252 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:26:48.909639 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:26:48.909927 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0810 03:26:48.910314 (+ 387us) server_negotiation.cc:299] Negotiation successful
0810 03:26:48.910591 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260810 03:26:48.916378 7320 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53870:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 03:26:48.917160 7320 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 03:26:48.926129 7542 tablet_service.cc:1467] Processing CreateTablet for tablet 7deff53544944a18987cd0a577ef00e9 (DEFAULT_TABLE table=test-table [id=03788b8ca67842d2b491bfe3cf6407d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:26:48.926681 7542 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7deff53544944a18987cd0a577ef00e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:26:48.926604 7742 tablet_service.cc:1467] Processing CreateTablet for tablet 7deff53544944a18987cd0a577ef00e9 (DEFAULT_TABLE table=test-table [id=03788b8ca67842d2b491bfe3cf6407d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:26:48.927040 7742 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7deff53544944a18987cd0a577ef00e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:26:48.933823 7946 tablet_service.cc:1467] Processing CreateTablet for tablet 7deff53544944a18987cd0a577ef00e9 (DEFAULT_TABLE table=test-table [id=03788b8ca67842d2b491bfe3cf6407d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:26:48.934345 7946 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7deff53544944a18987cd0a577ef00e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:26:48.937554 8202 tablet_bootstrap.cc:492] T 7deff53544944a18987cd0a577ef00e9 P db6b5c27fe7b480f96dbf4e5afd7b6b9: Bootstrap starting.
I20260810 03:26:48.940253 8202 tablet_bootstrap.cc:654] T 7deff53544944a18987cd0a577ef00e9 P db6b5c27fe7b480f96dbf4e5afd7b6b9: Neither blocks nor log segments found. Creating new log.
I20260810 03:26:48.944591 8202 tablet_bootstrap.cc:492] T 7deff53544944a18987cd0a577ef00e9 P db6b5c27fe7b480f96dbf4e5afd7b6b9: No bootstrap required, opened a new log
I20260810 03:26:48.946337 8202 ts_tablet_manager.cc:1397] T 7deff53544944a18987cd0a577ef00e9 P db6b5c27fe7b480f96dbf4e5afd7b6b9: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.000s
I20260810 03:26:48.947365 8202 raft_consensus.cc:348] T 7deff53544944a18987cd0a577ef00e9 P db6b5c27fe7b480f96dbf4e5afd7b6b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51a5b515913a4b10bd40695507e0fcca" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 35325 } } peers { permanent_uuid: "db6b5c27fe7b480f96dbf4e5afd7b6b9" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 33609 } } peers { permanent_uuid: "9a1b431e2e804faab2004da408cd20f8" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 40179 } }
I20260810 03:26:48.947809 8202 raft_consensus.cc:374] T 7deff53544944a18987cd0a577ef00e9 P db6b5c27fe7b480f96dbf4e5afd7b6b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:26:48.947950 8202 raft_consensus.cc:729] T 7deff53544944a18987cd0a577ef00e9 P db6b5c27fe7b480f96dbf4e5afd7b6b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db6b5c27fe7b480f96dbf4e5afd7b6b9, State: Initialized, Role: FOLLOWER
I20260810 03:26:48.948329 8202 consensus_queue.cc:260] T 7deff53544944a18987cd0a577ef00e9 P db6b5c27fe7b480f96dbf4e5afd7b6b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51a5b515913a4b10bd40695507e0fcca" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 35325 } } peers { permanent_uuid: "db6b5c27fe7b480f96dbf4e5afd7b6b9" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 33609 } } peers { permanent_uuid: "9a1b431e2e804faab2004da408cd20f8" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 40179 } }
I20260810 03:26:48.951633 8206 tablet_bootstrap.cc:492] T 7deff53544944a18987cd0a577ef00e9 P 9a1b431e2e804faab2004da408cd20f8: Bootstrap starting.
I20260810 03:26:48.954569 8206 tablet_bootstrap.cc:654] T 7deff53544944a18987cd0a577ef00e9 P 9a1b431e2e804faab2004da408cd20f8: Neither blocks nor log segments found. Creating new log.
I20260810 03:26:48.955400 8202 ts_tablet_manager.cc:1428] T 7deff53544944a18987cd0a577ef00e9 P db6b5c27fe7b480f96dbf4e5afd7b6b9: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20260810 03:26:48.957183 8204 tablet_bootstrap.cc:492] T 7deff53544944a18987cd0a577ef00e9 P 51a5b515913a4b10bd40695507e0fcca: Bootstrap starting.
I20260810 03:26:48.959148 8204 tablet_bootstrap.cc:654] T 7deff53544944a18987cd0a577ef00e9 P 51a5b515913a4b10bd40695507e0fcca: Neither blocks nor log segments found. Creating new log.
I20260810 03:26:48.963887 8204 tablet_bootstrap.cc:492] T 7deff53544944a18987cd0a577ef00e9 P 51a5b515913a4b10bd40695507e0fcca: No bootstrap required, opened a new log
I20260810 03:26:48.963917 8206 tablet_bootstrap.cc:492] T 7deff53544944a18987cd0a577ef00e9 P 9a1b431e2e804faab2004da408cd20f8: No bootstrap required, opened a new log
I20260810 03:26:48.964140 8206 ts_tablet_manager.cc:1397] T 7deff53544944a18987cd0a577ef00e9 P 9a1b431e2e804faab2004da408cd20f8: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.007s
I20260810 03:26:48.964129 8204 ts_tablet_manager.cc:1397] T 7deff53544944a18987cd0a577ef00e9 P 51a5b515913a4b10bd40695507e0fcca: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.002s
I20260810 03:26:48.965103 8206 raft_consensus.cc:348] T 7deff53544944a18987cd0a577ef00e9 P 9a1b431e2e804faab2004da408cd20f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51a5b515913a4b10bd40695507e0fcca" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 35325 } } peers { permanent_uuid: "db6b5c27fe7b480f96dbf4e5afd7b6b9" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 33609 } } peers { permanent_uuid: "9a1b431e2e804faab2004da408cd20f8" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 40179 } }
I20260810 03:26:48.965461 8206 raft_consensus.cc:374] T 7deff53544944a18987cd0a577ef00e9 P 9a1b431e2e804faab2004da408cd20f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:26:48.965631 8206 raft_consensus.cc:729] T 7deff53544944a18987cd0a577ef00e9 P 9a1b431e2e804faab2004da408cd20f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a1b431e2e804faab2004da408cd20f8, State: Initialized, Role: FOLLOWER
I20260810 03:26:48.966180 8206 consensus_queue.cc:260] T 7deff53544944a18987cd0a577ef00e9 P 9a1b431e2e804faab2004da408cd20f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51a5b515913a4b10bd40695507e0fcca" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 35325 } } peers { permanent_uuid: "db6b5c27fe7b480f96dbf4e5afd7b6b9" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 33609 } } peers { permanent_uuid: "9a1b431e2e804faab2004da408cd20f8" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 40179 } }
I20260810 03:26:48.965090 8204 raft_consensus.cc:348] T 7deff53544944a18987cd0a577ef00e9 P 51a5b515913a4b10bd40695507e0fcca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51a5b515913a4b10bd40695507e0fcca" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 35325 } } peers { permanent_uuid: "db6b5c27fe7b480f96dbf4e5afd7b6b9" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 33609 } } peers { permanent_uuid: "9a1b431e2e804faab2004da408cd20f8" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 40179 } }
I20260810 03:26:48.967152 8204 raft_consensus.cc:374] T 7deff53544944a18987cd0a577ef00e9 P 51a5b515913a4b10bd40695507e0fcca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:26:48.967316 8204 raft_consensus.cc:729] T 7deff53544944a18987cd0a577ef00e9 P 51a5b515913a4b10bd40695507e0fcca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51a5b515913a4b10bd40695507e0fcca, State: Initialized, Role: FOLLOWER
I20260810 03:26:48.967641 8204 consensus_queue.cc:260] T 7deff53544944a18987cd0a577ef00e9 P 51a5b515913a4b10bd40695507e0fcca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51a5b515913a4b10bd40695507e0fcca" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 35325 } } peers { permanent_uuid: "db6b5c27fe7b480f96dbf4e5afd7b6b9" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 33609 } } peers { permanent_uuid: "9a1b431e2e804faab2004da408cd20f8" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 40179 } }
I20260810 03:26:48.968420 8204 ts_tablet_manager.cc:1428] T 7deff53544944a18987cd0a577ef00e9 P 51a5b515913a4b10bd40695507e0fcca: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20260810 03:26:48.969530 8206 ts_tablet_manager.cc:1428] T 7deff53544944a18987cd0a577ef00e9 P 9a1b431e2e804faab2004da408cd20f8: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20260810 03:26:49.056865 8098 raft_consensus.cc:482] T 7deff53544944a18987cd0a577ef00e9 P db6b5c27fe7b480f96dbf4e5afd7b6b9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 03:26:49.057088 8098 raft_consensus.cc:504] T 7deff53544944a18987cd0a577ef00e9 P db6b5c27fe7b480f96dbf4e5afd7b6b9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51a5b515913a4b10bd40695507e0fcca" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 35325 } } peers { permanent_uuid: "db6b5c27fe7b480f96dbf4e5afd7b6b9" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 33609 } } peers { permanent_uuid: "9a1b431e2e804faab2004da408cd20f8" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 40179 } }
I20260810 03:26:49.057830 8098 leader_election.cc:288] T 7deff53544944a18987cd0a577ef00e9 P db6b5c27fe7b480f96dbf4e5afd7b6b9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51a5b515913a4b10bd40695507e0fcca (127.27.161.1:35325), 9a1b431e2e804faab2004da408cd20f8 (127.27.161.3:40179)
I20260810 03:26:49.058513 7569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7deff53544944a18987cd0a577ef00e9" candidate_uuid: "db6b5c27fe7b480f96dbf4e5afd7b6b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51a5b515913a4b10bd40695507e0fcca" is_pre_election: true
I20260810 03:26:49.058615 7977 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7deff53544944a18987cd0a577ef00e9" candidate_uuid: "db6b5c27fe7b480f96dbf4e5afd7b6b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a1b431e2e804faab2004da408cd20f8" is_pre_election: true
I20260810 03:26:49.058856 7977 raft_consensus.cc:2454] T 7deff53544944a18987cd0a577ef00e9 P 9a1b431e2e804faab2004da408cd20f8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db6b5c27fe7b480f96dbf4e5afd7b6b9 in term 0.
I20260810 03:26:49.058851 7569 raft_consensus.cc:2454] T 7deff53544944a18987cd0a577ef00e9 P 51a5b515913a4b10bd40695507e0fcca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db6b5c27fe7b480f96dbf4e5afd7b6b9 in term 0.
I20260810 03:26:49.059442 7674 leader_election.cc:302] T 7deff53544944a18987cd0a577ef00e9 P db6b5c27fe7b480f96dbf4e5afd7b6b9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a1b431e2e804faab2004da408cd20f8, db6b5c27fe7b480f96dbf4e5afd7b6b9; no voters:
I20260810 03:26:49.059860 8098 raft_consensus.cc:2781] T 7deff53544944a18987cd0a577ef00e9 P db6b5c27fe7b480f96dbf4e5afd7b6b9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 03:26:49.059979 8098 raft_consensus.cc:482] T 7deff53544944a18987cd0a577ef00e9 P db6b5c27fe7b480f96dbf4e5afd7b6b9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 03:26:49.060066 8098 raft_consensus.cc:3037] T 7deff53544944a18987cd0a577ef00e9 P db6b5c27fe7b480f96dbf4e5afd7b6b9 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:26:49.062304 8098 raft_consensus.cc:504] T 7deff53544944a18987cd0a577ef00e9 P db6b5c27fe7b480f96dbf4e5afd7b6b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51a5b515913a4b10bd40695507e0fcca" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 35325 } } peers { permanent_uuid: "db6b5c27fe7b480f96dbf4e5afd7b6b9" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 33609 } } peers { permanent_uuid: "9a1b431e2e804faab2004da408cd20f8" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 40179 } }
I20260810 03:26:49.062886 8098 leader_election.cc:288] T 7deff53544944a18987cd0a577ef00e9 P db6b5c27fe7b480f96dbf4e5afd7b6b9 [CANDIDATE]: Term 1 election: Requested vote from peers 51a5b515913a4b10bd40695507e0fcca (127.27.161.1:35325), 9a1b431e2e804faab2004da408cd20f8 (127.27.161.3:40179)
I20260810 03:26:49.063444 7569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7deff53544944a18987cd0a577ef00e9" candidate_uuid: "db6b5c27fe7b480f96dbf4e5afd7b6b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51a5b515913a4b10bd40695507e0fcca"
I20260810 03:26:49.063680 7569 raft_consensus.cc:3037] T 7deff53544944a18987cd0a577ef00e9 P 51a5b515913a4b10bd40695507e0fcca [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:26:49.063781 7977 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7deff53544944a18987cd0a577ef00e9" candidate_uuid: "db6b5c27fe7b480f96dbf4e5afd7b6b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a1b431e2e804faab2004da408cd20f8"
I20260810 03:26:49.064009 7977 raft_consensus.cc:3037] T 7deff53544944a18987cd0a577ef00e9 P 9a1b431e2e804faab2004da408cd20f8 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:26:49.066910 7569 raft_consensus.cc:2454] T 7deff53544944a18987cd0a577ef00e9 P 51a5b515913a4b10bd40695507e0fcca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db6b5c27fe7b480f96dbf4e5afd7b6b9 in term 1.
I20260810 03:26:49.067133 7977 raft_consensus.cc:2454] T 7deff53544944a18987cd0a577ef00e9 P 9a1b431e2e804faab2004da408cd20f8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db6b5c27fe7b480f96dbf4e5afd7b6b9 in term 1.
I20260810 03:26:49.067579 7675 leader_election.cc:302] T 7deff53544944a18987cd0a577ef00e9 P db6b5c27fe7b480f96dbf4e5afd7b6b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51a5b515913a4b10bd40695507e0fcca, db6b5c27fe7b480f96dbf4e5afd7b6b9; no voters:
I20260810 03:26:49.068188 8098 raft_consensus.cc:2781] T 7deff53544944a18987cd0a577ef00e9 P db6b5c27fe7b480f96dbf4e5afd7b6b9 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:26:49.069233 8098 raft_consensus.cc:686] T 7deff53544944a18987cd0a577ef00e9 P db6b5c27fe7b480f96dbf4e5afd7b6b9 [term 1 LEADER]: Becoming Leader. State: Replica: db6b5c27fe7b480f96dbf4e5afd7b6b9, State: Running, Role: LEADER
I20260810 03:26:49.069626 8098 consensus_queue.cc:237] T 7deff53544944a18987cd0a577ef00e9 P db6b5c27fe7b480f96dbf4e5afd7b6b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51a5b515913a4b10bd40695507e0fcca" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 35325 } } peers { permanent_uuid: "db6b5c27fe7b480f96dbf4e5afd7b6b9" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 33609 } } peers { permanent_uuid: "9a1b431e2e804faab2004da408cd20f8" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 40179 } }
I20260810 03:26:49.083009 7318 catalog_manager.cc:5495] T 7deff53544944a18987cd0a577ef00e9 P db6b5c27fe7b480f96dbf4e5afd7b6b9 reported cstate change: term changed from 0 to 1, leader changed from <none> to db6b5c27fe7b480f96dbf4e5afd7b6b9 (127.27.161.2). New cstate: current_term: 1 leader_uuid: "db6b5c27fe7b480f96dbf4e5afd7b6b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51a5b515913a4b10bd40695507e0fcca" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 35325 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "db6b5c27fe7b480f96dbf4e5afd7b6b9" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 33609 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9a1b431e2e804faab2004da408cd20f8" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 40179 } health_report { overall_health: UNKNOWN } } }
I20260810 03:26:49.109995 8216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:49.104624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36798 (local address 127.27.161.2:33609)
0810 03:26:49.105192 (+ 568us) server_negotiation.cc:206] Beginning negotiation
0810 03:26:49.105197 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:26:49.105212 (+ 15us) server_negotiation.cc:407] Connection header received
0810 03:26:49.105273 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:26:49.105277 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:26:49.105340 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:26:49.105432 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 03:26:49.105838 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:49.106676 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:49.108088 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:49.108512 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:49.108633 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:49.109128 (+ 495us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 03:26:49.109270 (+ 142us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 03:26:49.109351 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0810 03:26:49.109526 (+ 175us) server_negotiation.cc:299] Negotiation successful
0810 03:26:49.109619 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":445,"thread_start_us":278,"threads_started":1}
I20260810 03:26:49.113382 7569 raft_consensus.cc:1261] T 7deff53544944a18987cd0a577ef00e9 P 51a5b515913a4b10bd40695507e0fcca [term 1 FOLLOWER]: Refusing update from remote peer db6b5c27fe7b480f96dbf4e5afd7b6b9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:26:49.113404 7977 raft_consensus.cc:1261] T 7deff53544944a18987cd0a577ef00e9 P 9a1b431e2e804faab2004da408cd20f8 [term 1 FOLLOWER]: Refusing update from remote peer db6b5c27fe7b480f96dbf4e5afd7b6b9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:26:49.114199 8210 consensus_queue.cc:1035] T 7deff53544944a18987cd0a577ef00e9 P db6b5c27fe7b480f96dbf4e5afd7b6b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a1b431e2e804faab2004da408cd20f8" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 40179 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:26:49.114657 8098 consensus_queue.cc:1035] T 7deff53544944a18987cd0a577ef00e9 P db6b5c27fe7b480f96dbf4e5afd7b6b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51a5b515913a4b10bd40695507e0fcca" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 35325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:26:49.127180 7318 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53870:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 03:26:49.127473 7318 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53870:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 03:26:49.135565 7542 tablet_service.cc:1514] Processing DeleteTablet for tablet 7deff53544944a18987cd0a577ef00e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:26:49 UTC) from {username='oryx', principal='oryx/127.27.161.62@KRBTEST.COM'} at 127.0.0.1:38524
I20260810 03:26:49.135910 8182 tablet_replica.cc:330] T 7deff53544944a18987cd0a577ef00e9 P 51a5b515913a4b10bd40695507e0fcca: stopping tablet replica
I20260810 03:26:49.136111 8182 raft_consensus.cc:2229] T 7deff53544944a18987cd0a577ef00e9 P 51a5b515913a4b10bd40695507e0fcca [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 03:26:49.136098 7318 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6882776d888e4399b85872d4661cecf3: Sending DeleteTablet for 3 replicas of tablet 7deff53544944a18987cd0a577ef00e9
I20260810 03:26:49.136338 8182 raft_consensus.cc:2258] T 7deff53544944a18987cd0a577ef00e9 P 51a5b515913a4b10bd40695507e0fcca [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:26:49.136705 7946 tablet_service.cc:1514] Processing DeleteTablet for tablet 7deff53544944a18987cd0a577ef00e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:26:49 UTC) from {username='oryx', principal='oryx/127.27.161.62@KRBTEST.COM'} at 127.0.0.1:51190
I20260810 03:26:49.137008 8185 tablet_replica.cc:330] T 7deff53544944a18987cd0a577ef00e9 P 9a1b431e2e804faab2004da408cd20f8: stopping tablet replica
I20260810 03:26:49.137310 8185 raft_consensus.cc:2229] T 7deff53544944a18987cd0a577ef00e9 P 9a1b431e2e804faab2004da408cd20f8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 03:26:49.137370 7946 tablet_service.cc:1514] Processing DeleteTablet for tablet 7deff53544944a18987cd0a577ef00e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:26:49 UTC) from {username='oryx', principal='oryx/127.27.161.62@KRBTEST.COM'} at 127.0.0.1:51190
I20260810 03:26:49.137567 8185 raft_consensus.cc:2258] T 7deff53544944a18987cd0a577ef00e9 P 9a1b431e2e804faab2004da408cd20f8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:26:49.138221 7542 tablet_service.cc:1514] Processing DeleteTablet for tablet 7deff53544944a18987cd0a577ef00e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:26:49 UTC) from {username='oryx', principal='oryx/127.27.161.62@KRBTEST.COM'} at 127.0.0.1:38524
I20260810 03:26:49.138984 8185 ts_tablet_manager.cc:1905] T 7deff53544944a18987cd0a577ef00e9 P 9a1b431e2e804faab2004da408cd20f8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:26:49.139057 7742 tablet_service.cc:1514] Processing DeleteTablet for tablet 7deff53544944a18987cd0a577ef00e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:26:49 UTC) from {username='oryx', principal='oryx/127.27.161.62@KRBTEST.COM'} at 127.0.0.1:36816
I20260810 03:26:49.139115 7741 tablet_service.cc:1514] Processing DeleteTablet for tablet 7deff53544944a18987cd0a577ef00e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:26:49 UTC) from {username='oryx', principal='oryx/127.27.161.62@KRBTEST.COM'} at 127.0.0.1:36816
I20260810 03:26:49.138994 7318 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53870:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 03:26:49.139339 8183 tablet_replica.cc:330] T 7deff53544944a18987cd0a577ef00e9 P db6b5c27fe7b480f96dbf4e5afd7b6b9: stopping tablet replica
I20260810 03:26:49.139525 8183 raft_consensus.cc:2229] T 7deff53544944a18987cd0a577ef00e9 P db6b5c27fe7b480f96dbf4e5afd7b6b9 [term 1 LEADER]: Raft consensus shutting down.
W20260810 03:26:49.139632 7318 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 03:26:49.139855 8183 raft_consensus.cc:2258] T 7deff53544944a18987cd0a577ef00e9 P db6b5c27fe7b480f96dbf4e5afd7b6b9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:26:49.141217 8183 ts_tablet_manager.cc:1905] T 7deff53544944a18987cd0a577ef00e9 P db6b5c27fe7b480f96dbf4e5afd7b6b9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:26:49.144441 8182 ts_tablet_manager.cc:1905] T 7deff53544944a18987cd0a577ef00e9 P 51a5b515913a4b10bd40695507e0fcca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:26:49.145936 8185 ts_tablet_manager.cc:1918] T 7deff53544944a18987cd0a577ef00e9 P 9a1b431e2e804faab2004da408cd20f8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 03:26:49.146162 8185 log.cc:1196] T 7deff53544944a18987cd0a577ef00e9 P 9a1b431e2e804faab2004da408cd20f8: Deleting WAL directory at /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786332336319544-28292-0/minicluster-data/ts-2/wal/wals/7deff53544944a18987cd0a577ef00e9
I20260810 03:26:49.146626 8185 ts_tablet_manager.cc:1939] T 7deff53544944a18987cd0a577ef00e9 P 9a1b431e2e804faab2004da408cd20f8: Deleting consensus metadata
I20260810 03:26:49.148247 7542 tablet_service.cc:1467] Processing CreateTablet for tablet 53995bf7e4344ec9826f2f471f07bb80 (DEFAULT_TABLE table=test-table [id=340f040ec02742a094b77c86e7941f94]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:26:49.148764 7542 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53995bf7e4344ec9826f2f471f07bb80. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:26:49.151211 7741 tablet_service.cc:1467] Processing CreateTablet for tablet 53995bf7e4344ec9826f2f471f07bb80 (DEFAULT_TABLE table=test-table [id=340f040ec02742a094b77c86e7941f94]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:26:49.151746 7741 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53995bf7e4344ec9826f2f471f07bb80. 1 dirs total, 0 dirs full, 0 dirs failed
W20260810 03:26:49.154546 7293 catalog_manager.cc:4808] TS 9a1b431e2e804faab2004da408cd20f8 (127.27.161.3:40179): delete failed for tablet 7deff53544944a18987cd0a577ef00e9 because the tablet was not found. No further retry: Not found: Tablet not found: 7deff53544944a18987cd0a577ef00e9
I20260810 03:26:49.155275 7946 tablet_service.cc:1467] Processing CreateTablet for tablet 53995bf7e4344ec9826f2f471f07bb80 (DEFAULT_TABLE table=test-table [id=340f040ec02742a094b77c86e7941f94]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:26:49.155579 8183 ts_tablet_manager.cc:1918] T 7deff53544944a18987cd0a577ef00e9 P db6b5c27fe7b480f96dbf4e5afd7b6b9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 03:26:49.155741 8183 log.cc:1196] T 7deff53544944a18987cd0a577ef00e9 P db6b5c27fe7b480f96dbf4e5afd7b6b9: Deleting WAL directory at /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786332336319544-28292-0/minicluster-data/ts-1/wal/wals/7deff53544944a18987cd0a577ef00e9
I20260810 03:26:49.155766 7946 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53995bf7e4344ec9826f2f471f07bb80. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:26:49.156203 8183 ts_tablet_manager.cc:1939] T 7deff53544944a18987cd0a577ef00e9 P db6b5c27fe7b480f96dbf4e5afd7b6b9: Deleting consensus metadata
I20260810 03:26:49.158384 8182 ts_tablet_manager.cc:1918] T 7deff53544944a18987cd0a577ef00e9 P 51a5b515913a4b10bd40695507e0fcca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 03:26:49.158587 8182 log.cc:1196] T 7deff53544944a18987cd0a577ef00e9 P 51a5b515913a4b10bd40695507e0fcca: Deleting WAL directory at /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786332336319544-28292-0/minicluster-data/ts-0/wal/wals/7deff53544944a18987cd0a577ef00e9
I20260810 03:26:49.159050 8182 ts_tablet_manager.cc:1939] T 7deff53544944a18987cd0a577ef00e9 P 51a5b515913a4b10bd40695507e0fcca: Deleting consensus metadata
I20260810 03:26:49.159088 8202 tablet_bootstrap.cc:492] T 53995bf7e4344ec9826f2f471f07bb80 P db6b5c27fe7b480f96dbf4e5afd7b6b9: Bootstrap starting.
I20260810 03:26:49.160765 8202 tablet_bootstrap.cc:654] T 53995bf7e4344ec9826f2f471f07bb80 P db6b5c27fe7b480f96dbf4e5afd7b6b9: Neither blocks nor log segments found. Creating new log.
W20260810 03:26:49.160698 7294 catalog_manager.cc:4808] TS 51a5b515913a4b10bd40695507e0fcca (127.27.161.1:35325): delete failed for tablet 7deff53544944a18987cd0a577ef00e9 because the tablet was not found. No further retry: Not found: Tablet not found: 7deff53544944a18987cd0a577ef00e9
W20260810 03:26:49.160979 7293 catalog_manager.cc:4808] TS db6b5c27fe7b480f96dbf4e5afd7b6b9 (127.27.161.2:33609): delete failed for tablet 7deff53544944a18987cd0a577ef00e9 because the tablet was not found. No further retry: Not found: Tablet not found: 7deff53544944a18987cd0a577ef00e9
I20260810 03:26:49.162811 8202 tablet_bootstrap.cc:492] T 53995bf7e4344ec9826f2f471f07bb80 P db6b5c27fe7b480f96dbf4e5afd7b6b9: No bootstrap required, opened a new log
I20260810 03:26:49.162950 8202 ts_tablet_manager.cc:1397] T 53995bf7e4344ec9826f2f471f07bb80 P db6b5c27fe7b480f96dbf4e5afd7b6b9: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260810 03:26:49.163583 8202 raft_consensus.cc:348] T 53995bf7e4344ec9826f2f471f07bb80 P db6b5c27fe7b480f96dbf4e5afd7b6b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51a5b515913a4b10bd40695507e0fcca" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 35325 } } peers { permanent_uuid: "db6b5c27fe7b480f96dbf4e5afd7b6b9" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 33609 } } peers { permanent_uuid: "9a1b431e2e804faab2004da408cd20f8" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 40179 } }
I20260810 03:26:49.163825 8202 raft_consensus.cc:374] T 53995bf7e4344ec9826f2f471f07bb80 P db6b5c27fe7b480f96dbf4e5afd7b6b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:26:49.163911 8202 raft_consensus.cc:729] T 53995bf7e4344ec9826f2f471f07bb80 P db6b5c27fe7b480f96dbf4e5afd7b6b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db6b5c27fe7b480f96dbf4e5afd7b6b9, State: Initialized, Role: FOLLOWER
I20260810 03:26:49.164165 8202 consensus_queue.cc:260] T 53995bf7e4344ec9826f2f471f07bb80 P db6b5c27fe7b480f96dbf4e5afd7b6b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51a5b515913a4b10bd40695507e0fcca" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 35325 } } peers { permanent_uuid: "db6b5c27fe7b480f96dbf4e5afd7b6b9" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 33609 } } peers { permanent_uuid: "9a1b431e2e804faab2004da408cd20f8" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 40179 } }
I20260810 03:26:49.164912 8202 ts_tablet_manager.cc:1428] T 53995bf7e4344ec9826f2f471f07bb80 P db6b5c27fe7b480f96dbf4e5afd7b6b9: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260810 03:26:49.168226 8204 tablet_bootstrap.cc:492] T 53995bf7e4344ec9826f2f471f07bb80 P 51a5b515913a4b10bd40695507e0fcca: Bootstrap starting.
I20260810 03:26:49.169771 8206 tablet_bootstrap.cc:492] T 53995bf7e4344ec9826f2f471f07bb80 P 9a1b431e2e804faab2004da408cd20f8: Bootstrap starting.
I20260810 03:26:49.170006 8204 tablet_bootstrap.cc:654] T 53995bf7e4344ec9826f2f471f07bb80 P 51a5b515913a4b10bd40695507e0fcca: Neither blocks nor log segments found. Creating new log.
I20260810 03:26:49.171681 8206 tablet_bootstrap.cc:654] T 53995bf7e4344ec9826f2f471f07bb80 P 9a1b431e2e804faab2004da408cd20f8: Neither blocks nor log segments found. Creating new log.
I20260810 03:26:49.171963 8204 tablet_bootstrap.cc:492] T 53995bf7e4344ec9826f2f471f07bb80 P 51a5b515913a4b10bd40695507e0fcca: No bootstrap required, opened a new log
I20260810 03:26:49.172158 8204 ts_tablet_manager.cc:1397] T 53995bf7e4344ec9826f2f471f07bb80 P 51a5b515913a4b10bd40695507e0fcca: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20260810 03:26:49.172900 8204 raft_consensus.cc:348] T 53995bf7e4344ec9826f2f471f07bb80 P 51a5b515913a4b10bd40695507e0fcca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51a5b515913a4b10bd40695507e0fcca" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 35325 } } peers { permanent_uuid: "db6b5c27fe7b480f96dbf4e5afd7b6b9" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 33609 } } peers { permanent_uuid: "9a1b431e2e804faab2004da408cd20f8" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 40179 } }
I20260810 03:26:49.173169 8204 raft_consensus.cc:374] T 53995bf7e4344ec9826f2f471f07bb80 P 51a5b515913a4b10bd40695507e0fcca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:26:49.173260 8204 raft_consensus.cc:729] T 53995bf7e4344ec9826f2f471f07bb80 P 51a5b515913a4b10bd40695507e0fcca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51a5b515913a4b10bd40695507e0fcca, State: Initialized, Role: FOLLOWER
I20260810 03:26:49.173556 8204 consensus_queue.cc:260] T 53995bf7e4344ec9826f2f471f07bb80 P 51a5b515913a4b10bd40695507e0fcca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51a5b515913a4b10bd40695507e0fcca" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 35325 } } peers { permanent_uuid: "db6b5c27fe7b480f96dbf4e5afd7b6b9" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 33609 } } peers { permanent_uuid: "9a1b431e2e804faab2004da408cd20f8" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 40179 } }
I20260810 03:26:49.174921 8204 ts_tablet_manager.cc:1428] T 53995bf7e4344ec9826f2f471f07bb80 P 51a5b515913a4b10bd40695507e0fcca: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260810 03:26:49.175132 8206 tablet_bootstrap.cc:492] T 53995bf7e4344ec9826f2f471f07bb80 P 9a1b431e2e804faab2004da408cd20f8: No bootstrap required, opened a new log
I20260810 03:26:49.175328 8206 ts_tablet_manager.cc:1397] T 53995bf7e4344ec9826f2f471f07bb80 P 9a1b431e2e804faab2004da408cd20f8: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260810 03:26:49.176160 8206 raft_consensus.cc:348] T 53995bf7e4344ec9826f2f471f07bb80 P 9a1b431e2e804faab2004da408cd20f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51a5b515913a4b10bd40695507e0fcca" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 35325 } } peers { permanent_uuid: "db6b5c27fe7b480f96dbf4e5afd7b6b9" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 33609 } } peers { permanent_uuid: "9a1b431e2e804faab2004da408cd20f8" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 40179 } }
I20260810 03:26:49.176476 8206 raft_consensus.cc:374] T 53995bf7e4344ec9826f2f471f07bb80 P 9a1b431e2e804faab2004da408cd20f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:26:49.176584 8206 raft_consensus.cc:729] T 53995bf7e4344ec9826f2f471f07bb80 P 9a1b431e2e804faab2004da408cd20f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a1b431e2e804faab2004da408cd20f8, State: Initialized, Role: FOLLOWER
I20260810 03:26:49.176851 8206 consensus_queue.cc:260] T 53995bf7e4344ec9826f2f471f07bb80 P 9a1b431e2e804faab2004da408cd20f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51a5b515913a4b10bd40695507e0fcca" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 35325 } } peers { permanent_uuid: "db6b5c27fe7b480f96dbf4e5afd7b6b9" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 33609 } } peers { permanent_uuid: "9a1b431e2e804faab2004da408cd20f8" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 40179 } }
I20260810 03:26:49.177446 8206 ts_tablet_manager.cc:1428] T 53995bf7e4344ec9826f2f471f07bb80 P 9a1b431e2e804faab2004da408cd20f8: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260810 03:26:49.221602 8180 raft_consensus.cc:482] T 53995bf7e4344ec9826f2f471f07bb80 P 9a1b431e2e804faab2004da408cd20f8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 03:26:49.221863 8180 raft_consensus.cc:504] T 53995bf7e4344ec9826f2f471f07bb80 P 9a1b431e2e804faab2004da408cd20f8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51a5b515913a4b10bd40695507e0fcca" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 35325 } } peers { permanent_uuid: "db6b5c27fe7b480f96dbf4e5afd7b6b9" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 33609 } } peers { permanent_uuid: "9a1b431e2e804faab2004da408cd20f8" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 40179 } }
I20260810 03:26:49.222601 8180 leader_election.cc:288] T 53995bf7e4344ec9826f2f471f07bb80 P 9a1b431e2e804faab2004da408cd20f8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51a5b515913a4b10bd40695507e0fcca (127.27.161.1:35325), db6b5c27fe7b480f96dbf4e5afd7b6b9 (127.27.161.2:33609)
I20260810 03:26:49.223297 7569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53995bf7e4344ec9826f2f471f07bb80" candidate_uuid: "9a1b431e2e804faab2004da408cd20f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51a5b515913a4b10bd40695507e0fcca" is_pre_election: true
I20260810 03:26:49.223276 7777 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53995bf7e4344ec9826f2f471f07bb80" candidate_uuid: "9a1b431e2e804faab2004da408cd20f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db6b5c27fe7b480f96dbf4e5afd7b6b9" is_pre_election: true
I20260810 03:26:49.223551 7569 raft_consensus.cc:2454] T 53995bf7e4344ec9826f2f471f07bb80 P 51a5b515913a4b10bd40695507e0fcca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a1b431e2e804faab2004da408cd20f8 in term 0.
I20260810 03:26:49.223567 7777 raft_consensus.cc:2454] T 53995bf7e4344ec9826f2f471f07bb80 P db6b5c27fe7b480f96dbf4e5afd7b6b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a1b431e2e804faab2004da408cd20f8 in term 0.
I20260810 03:26:49.224051 7877 leader_election.cc:302] T 53995bf7e4344ec9826f2f471f07bb80 P 9a1b431e2e804faab2004da408cd20f8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a1b431e2e804faab2004da408cd20f8, db6b5c27fe7b480f96dbf4e5afd7b6b9; no voters:
I20260810 03:26:49.224367 8180 raft_consensus.cc:2781] T 53995bf7e4344ec9826f2f471f07bb80 P 9a1b431e2e804faab2004da408cd20f8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 03:26:49.224504 8180 raft_consensus.cc:482] T 53995bf7e4344ec9826f2f471f07bb80 P 9a1b431e2e804faab2004da408cd20f8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 03:26:49.224607 8180 raft_consensus.cc:3037] T 53995bf7e4344ec9826f2f471f07bb80 P 9a1b431e2e804faab2004da408cd20f8 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:26:49.227304 8180 raft_consensus.cc:504] T 53995bf7e4344ec9826f2f471f07bb80 P 9a1b431e2e804faab2004da408cd20f8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51a5b515913a4b10bd40695507e0fcca" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 35325 } } peers { permanent_uuid: "db6b5c27fe7b480f96dbf4e5afd7b6b9" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 33609 } } peers { permanent_uuid: "9a1b431e2e804faab2004da408cd20f8" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 40179 } }
I20260810 03:26:49.227923 8180 leader_election.cc:288] T 53995bf7e4344ec9826f2f471f07bb80 P 9a1b431e2e804faab2004da408cd20f8 [CANDIDATE]: Term 1 election: Requested vote from peers 51a5b515913a4b10bd40695507e0fcca (127.27.161.1:35325), db6b5c27fe7b480f96dbf4e5afd7b6b9 (127.27.161.2:33609)
I20260810 03:26:49.228448 7569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53995bf7e4344ec9826f2f471f07bb80" candidate_uuid: "9a1b431e2e804faab2004da408cd20f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51a5b515913a4b10bd40695507e0fcca"
I20260810 03:26:49.228709 7569 raft_consensus.cc:3037] T 53995bf7e4344ec9826f2f471f07bb80 P 51a5b515913a4b10bd40695507e0fcca [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:26:49.228711 7777 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53995bf7e4344ec9826f2f471f07bb80" candidate_uuid: "9a1b431e2e804faab2004da408cd20f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db6b5c27fe7b480f96dbf4e5afd7b6b9"
I20260810 03:26:49.228953 7777 raft_consensus.cc:3037] T 53995bf7e4344ec9826f2f471f07bb80 P db6b5c27fe7b480f96dbf4e5afd7b6b9 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:26:49.231796 7569 raft_consensus.cc:2454] T 53995bf7e4344ec9826f2f471f07bb80 P 51a5b515913a4b10bd40695507e0fcca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a1b431e2e804faab2004da408cd20f8 in term 1.
I20260810 03:26:49.231931 7777 raft_consensus.cc:2454] T 53995bf7e4344ec9826f2f471f07bb80 P db6b5c27fe7b480f96dbf4e5afd7b6b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a1b431e2e804faab2004da408cd20f8 in term 1.
I20260810 03:26:49.232326 7878 leader_election.cc:302] T 53995bf7e4344ec9826f2f471f07bb80 P 9a1b431e2e804faab2004da408cd20f8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51a5b515913a4b10bd40695507e0fcca, 9a1b431e2e804faab2004da408cd20f8; no voters:
I20260810 03:26:49.232721 8180 raft_consensus.cc:2781] T 53995bf7e4344ec9826f2f471f07bb80 P 9a1b431e2e804faab2004da408cd20f8 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:26:49.233197 8180 raft_consensus.cc:686] T 53995bf7e4344ec9826f2f471f07bb80 P 9a1b431e2e804faab2004da408cd20f8 [term 1 LEADER]: Becoming Leader. State: Replica: 9a1b431e2e804faab2004da408cd20f8, State: Running, Role: LEADER
I20260810 03:26:49.233755 8180 consensus_queue.cc:237] T 53995bf7e4344ec9826f2f471f07bb80 P 9a1b431e2e804faab2004da408cd20f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51a5b515913a4b10bd40695507e0fcca" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 35325 } } peers { permanent_uuid: "db6b5c27fe7b480f96dbf4e5afd7b6b9" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 33609 } } peers { permanent_uuid: "9a1b431e2e804faab2004da408cd20f8" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 40179 } }
I20260810 03:26:49.236711 7317 catalog_manager.cc:5495] T 53995bf7e4344ec9826f2f471f07bb80 P 9a1b431e2e804faab2004da408cd20f8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9a1b431e2e804faab2004da408cd20f8 (127.27.161.3). New cstate: current_term: 1 leader_uuid: "9a1b431e2e804faab2004da408cd20f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51a5b515913a4b10bd40695507e0fcca" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 35325 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "db6b5c27fe7b480f96dbf4e5afd7b6b9" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 33609 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9a1b431e2e804faab2004da408cd20f8" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 40179 } health_report { overall_health: HEALTHY } } }
I20260810 03:26:49.291261 8234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:49.278006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47922 (local address 127.27.161.3:40179)
0810 03:26:49.284910 (+ 6904us) server_negotiation.cc:206] Beginning negotiation
0810 03:26:49.284917 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:26:49.284943 (+ 26us) server_negotiation.cc:407] Connection header received
0810 03:26:49.285024 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:26:49.285030 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:26:49.285112 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:26:49.285244 (+ 132us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 03:26:49.285902 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:49.287193 (+ 1291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:49.288972 (+ 1779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:49.289684 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:49.289813 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:49.290371 (+ 558us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 03:26:49.290507 (+ 136us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 03:26:49.290594 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0810 03:26:49.290762 (+ 168us) server_negotiation.cc:299] Negotiation successful
0810 03:26:49.290852 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6746,"thread_start_us":6581,"threads_started":1}
I20260810 03:26:49.305203 7777 raft_consensus.cc:1261] T 53995bf7e4344ec9826f2f471f07bb80 P db6b5c27fe7b480f96dbf4e5afd7b6b9 [term 1 FOLLOWER]: Refusing update from remote peer 9a1b431e2e804faab2004da408cd20f8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:26:49.305491 7569 raft_consensus.cc:1261] T 53995bf7e4344ec9826f2f471f07bb80 P 51a5b515913a4b10bd40695507e0fcca [term 1 FOLLOWER]: Refusing update from remote peer 9a1b431e2e804faab2004da408cd20f8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:26:49.305830 8180 consensus_queue.cc:1035] T 53995bf7e4344ec9826f2f471f07bb80 P 9a1b431e2e804faab2004da408cd20f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db6b5c27fe7b480f96dbf4e5afd7b6b9" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 33609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:26:49.306232 8227 consensus_queue.cc:1035] T 53995bf7e4344ec9826f2f471f07bb80 P 9a1b431e2e804faab2004da408cd20f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51a5b515913a4b10bd40695507e0fcca" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 35325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:26:49.355322 7318 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53870:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 03:26:49.355567 7318 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53870:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 03:26:49.358817 7318 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6882776d888e4399b85872d4661cecf3: Sending DeleteTablet for 3 replicas of tablet 53995bf7e4344ec9826f2f471f07bb80
I20260810 03:26:49.359663 7542 tablet_service.cc:1514] Processing DeleteTablet for tablet 53995bf7e4344ec9826f2f471f07bb80 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:26:49 UTC) from {username='oryx', principal='oryx/127.27.161.62@KRBTEST.COM'} at 127.0.0.1:38524
I20260810 03:26:49.359668 7741 tablet_service.cc:1514] Processing DeleteTablet for tablet 53995bf7e4344ec9826f2f471f07bb80 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:26:49 UTC) from {username='oryx', principal='oryx/127.27.161.62@KRBTEST.COM'} at 127.0.0.1:36816
I20260810 03:26:49.359992 8182 tablet_replica.cc:330] T 53995bf7e4344ec9826f2f471f07bb80 P 51a5b515913a4b10bd40695507e0fcca: stopping tablet replica
I20260810 03:26:49.360001 8183 tablet_replica.cc:330] T 53995bf7e4344ec9826f2f471f07bb80 P db6b5c27fe7b480f96dbf4e5afd7b6b9: stopping tablet replica
I20260810 03:26:49.360154 8182 raft_consensus.cc:2229] T 53995bf7e4344ec9826f2f471f07bb80 P 51a5b515913a4b10bd40695507e0fcca [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 03:26:49.360196 8183 raft_consensus.cc:2229] T 53995bf7e4344ec9826f2f471f07bb80 P db6b5c27fe7b480f96dbf4e5afd7b6b9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 03:26:49.360356 8182 raft_consensus.cc:2258] T 53995bf7e4344ec9826f2f471f07bb80 P 51a5b515913a4b10bd40695507e0fcca [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:26:49.360420 8183 raft_consensus.cc:2258] T 53995bf7e4344ec9826f2f471f07bb80 P db6b5c27fe7b480f96dbf4e5afd7b6b9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:26:49.360929 7946 tablet_service.cc:1514] Processing DeleteTablet for tablet 53995bf7e4344ec9826f2f471f07bb80 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:26:49 UTC) from {username='oryx', principal='oryx/127.27.161.62@KRBTEST.COM'} at 127.0.0.1:51190
I20260810 03:26:49.361052 28292 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iBH5/build/asan/bin/kudu pid 7436
I20260810 03:26:49.361644 8185 tablet_replica.cc:330] T 53995bf7e4344ec9826f2f471f07bb80 P 9a1b431e2e804faab2004da408cd20f8: stopping tablet replica
I20260810 03:26:49.361840 8185 raft_consensus.cc:2229] T 53995bf7e4344ec9826f2f471f07bb80 P 9a1b431e2e804faab2004da408cd20f8 [term 1 LEADER]: Raft consensus shutting down.
I20260810 03:26:49.362095 8182 ts_tablet_manager.cc:1905] T 53995bf7e4344ec9826f2f471f07bb80 P 51a5b515913a4b10bd40695507e0fcca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:26:49.362238 8185 raft_consensus.cc:2258] T 53995bf7e4344ec9826f2f471f07bb80 P 9a1b431e2e804faab2004da408cd20f8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:26:49.363307 8183 ts_tablet_manager.cc:1905] T 53995bf7e4344ec9826f2f471f07bb80 P db6b5c27fe7b480f96dbf4e5afd7b6b9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:26:49.366710 8185 ts_tablet_manager.cc:1905] T 53995bf7e4344ec9826f2f471f07bb80 P 9a1b431e2e804faab2004da408cd20f8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:26:49.369078 8183 ts_tablet_manager.cc:1918] T 53995bf7e4344ec9826f2f471f07bb80 P db6b5c27fe7b480f96dbf4e5afd7b6b9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 03:26:49.369220 8183 log.cc:1196] T 53995bf7e4344ec9826f2f471f07bb80 P db6b5c27fe7b480f96dbf4e5afd7b6b9: Deleting WAL directory at /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786332336319544-28292-0/minicluster-data/ts-1/wal/wals/53995bf7e4344ec9826f2f471f07bb80
I20260810 03:26:49.369691 8183 ts_tablet_manager.cc:1939] T 53995bf7e4344ec9826f2f471f07bb80 P db6b5c27fe7b480f96dbf4e5afd7b6b9: Deleting consensus metadata
I20260810 03:26:49.370172 8182 ts_tablet_manager.cc:1918] T 53995bf7e4344ec9826f2f471f07bb80 P 51a5b515913a4b10bd40695507e0fcca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 03:26:49.370353 8182 log.cc:1196] T 53995bf7e4344ec9826f2f471f07bb80 P 51a5b515913a4b10bd40695507e0fcca: Deleting WAL directory at /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786332336319544-28292-0/minicluster-data/ts-0/wal/wals/53995bf7e4344ec9826f2f471f07bb80
I20260810 03:26:49.370814 8182 ts_tablet_manager.cc:1939] T 53995bf7e4344ec9826f2f471f07bb80 P 51a5b515913a4b10bd40695507e0fcca: Deleting consensus metadata
I20260810 03:26:49.371104 7293 catalog_manager.cc:4841] TS db6b5c27fe7b480f96dbf4e5afd7b6b9 (127.27.161.2:33609): tablet 53995bf7e4344ec9826f2f471f07bb80 (table test-table [id=340f040ec02742a094b77c86e7941f94]) successfully deleted
I20260810 03:26:49.372171 7294 catalog_manager.cc:4841] TS 51a5b515913a4b10bd40695507e0fcca (127.27.161.1:35325): tablet 53995bf7e4344ec9826f2f471f07bb80 (table test-table [id=340f040ec02742a094b77c86e7941f94]) successfully deleted
I20260810 03:26:49.373636 8185 ts_tablet_manager.cc:1918] T 53995bf7e4344ec9826f2f471f07bb80 P 9a1b431e2e804faab2004da408cd20f8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 03:26:49.373772 8185 log.cc:1196] T 53995bf7e4344ec9826f2f471f07bb80 P 9a1b431e2e804faab2004da408cd20f8: Deleting WAL directory at /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786332336319544-28292-0/minicluster-data/ts-2/wal/wals/53995bf7e4344ec9826f2f471f07bb80
I20260810 03:26:49.374140 8185 ts_tablet_manager.cc:1939] T 53995bf7e4344ec9826f2f471f07bb80 P 9a1b431e2e804faab2004da408cd20f8: Deleting consensus metadata
I20260810 03:26:49.375222 7293 catalog_manager.cc:4841] TS 9a1b431e2e804faab2004da408cd20f8 (127.27.161.3:40179): tablet 53995bf7e4344ec9826f2f471f07bb80 (table test-table [id=340f040ec02742a094b77c86e7941f94]) successfully deleted
Aug 10 03:26:49 dist-test-slave-vpvm krb5kdc[7252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332408, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.27.161.1@KRBTEST.COM
I20260810 03:26:49.387210 8244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:49.370365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51838 (local address 127.27.161.1:35325)
0810 03:26:49.371816 (+ 1451us) server_negotiation.cc:206] Beginning negotiation
0810 03:26:49.371823 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:26:49.372961 (+ 1138us) server_negotiation.cc:407] Connection header received
0810 03:26:49.373096 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:26:49.373102 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:26:49.373196 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:26:49.373316 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:26:49.374693 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:49.375601 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:49.376877 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:49.377300 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:49.377396 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:49.381336 (+ 3940us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:26:49.381356 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:26:49.381374 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:26:49.381416 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:26:49.384344 (+ 2928us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:26:49.385049 (+ 705us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:26:49.385058 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:26:49.385061 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:26:49.385218 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:26:49.385602 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:26:49.385606 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:26:49.385608 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:26:49.386124 (+ 516us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:26:49.386492 (+ 368us) server_negotiation.cc:1035] Waiting for connection context
0810 03:26:49.386670 (+ 178us) server_negotiation.cc:299] Negotiation successful
0810 03:26:49.386873 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1305,"thread_start_us":4608,"threads_started":1}
I20260810 03:26:49.387876 7639 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:26:49.514235 28292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iBH5/build/asan/bin/kudu with pid 7436
I20260810 03:26:49.547397 28292 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iBH5/build/asan/bin/kudu pid 7650
Aug 10 03:26:49 dist-test-slave-vpvm krb5kdc[7252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332408, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.27.161.2@KRBTEST.COM
I20260810 03:26:49.562918 8216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:49.548169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36812 (local address 127.27.161.2:33609)
0810 03:26:49.548358 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0810 03:26:49.548362 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:26:49.548382 (+ 20us) server_negotiation.cc:407] Connection header received
0810 03:26:49.548505 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:26:49.548509 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:26:49.548575 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:26:49.548659 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:26:49.550120 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:49.550934 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:49.552269 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:49.552707 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:49.552817 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:49.557367 (+ 4550us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:26:49.557387 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:26:49.557405 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:26:49.557447 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:26:49.560383 (+ 2936us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:26:49.561083 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:26:49.561090 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:26:49.561092 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:26:49.561237 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:26:49.561529 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:26:49.561533 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:26:49.561535 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:26:49.561919 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:26:49.562225 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0810 03:26:49.562401 (+ 176us) server_negotiation.cc:299] Negotiation successful
0810 03:26:49.562617 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260810 03:26:49.563598 7836 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:26:49.689604 28292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iBH5/build/asan/bin/kudu with pid 7650
I20260810 03:26:49.716516 28292 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iBH5/build/asan/bin/kudu pid 7851
Aug 10 03:26:49 dist-test-slave-vpvm krb5kdc[7252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332408, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.27.161.3@KRBTEST.COM
I20260810 03:26:49.731499 8234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:49.717226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47932 (local address 127.27.161.3:40179)
0810 03:26:49.717385 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0810 03:26:49.717390 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:26:49.717458 (+ 68us) server_negotiation.cc:407] Connection header received
0810 03:26:49.717603 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:26:49.717606 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:26:49.717683 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:26:49.717767 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:26:49.719047 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:49.719833 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:49.721003 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:49.721398 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:49.721492 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:49.725801 (+ 4309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:26:49.725821 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:26:49.725840 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:26:49.725886 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:26:49.728675 (+ 2789us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:26:49.729344 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:26:49.729350 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:26:49.729353 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:26:49.729483 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:26:49.729930 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:26:49.729935 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:26:49.729937 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:26:49.730358 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:26:49.730630 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0810 03:26:49.731015 (+ 385us) server_negotiation.cc:299] Negotiation successful
0810 03:26:49.731211 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260810 03:26:49.732261 8046 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:26:49.857997 28292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iBH5/build/asan/bin/kudu with pid 7851
I20260810 03:26:49.891067 28292 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iBH5/build/asan/bin/kudu pid 7270
I20260810 03:26:49.892225 7389 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:26:50.014560 28292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iBH5/build/asan/bin/kudu with pid 7270
2026-08-10T03:26:50Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (7138 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786332336319544-28292-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 03:26:50 dist-test-slave-vpvm krb5kdc[8253](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 03:26:50 dist-test-slave-vpvm krb5kdc[8253](info): set up 2 sockets
Aug 10 03:26:50 dist-test-slave-vpvm krb5kdc[8253](info): commencing operation
krb5kdc: starting...
W20260810 03:26:51.277544 28292 mini_kdc.cc:121] Time spent starting KDC: real 1.207s user 0.000s sys 0.046s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786332336319544-28292-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786332336319544-28292-0/krb5kdc/test-user.keytab.
Aug 10 03:26:51 dist-test-slave-vpvm krb5kdc[8253](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332411, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T03:26:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T03:26:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.161.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.161.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.161.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.62.keytab.
Entry for principal kudu/127.27.161.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.62.keytab.
I20260810 03:26:51.571696 28292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iBH5/build/asan/bin/kudu
/tmp/dist-test-task24iBH5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786332336319544-28292-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786332336319544-28292-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786332336319544-28292-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786332336319544-28292-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.161.62:38505
--webserver_interface=127.27.161.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.62.keytab
--principal=kudu/127.27.161.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.161.20:41639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.161.62:38505
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786332336319544-28292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786332336319544-28292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786332336319544-28292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:26:51.940451 8271 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:26:51.940831 8271 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:26:51.941001 8271 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:26:51.948823 8271 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:26:51.948995 8271 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:26:51.949072 8271 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:26:51.949186 8271 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:26:51.949246 8271 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:26:51.959583 8271 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.161.20:41639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786332336319544-28292-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786332336319544-28292-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.161.62:38505
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.161.62:38505
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786332336319544-28292-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.161.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786332336319544-28292-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:26:51.961094 8271 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:26:51.962512 8271 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:26:51.975556 8282 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:26:51.975629 8279 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:26:51.975629 8280 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:26:51.976263 8271 server_base.cc:1034] running on GCE node
I20260810 03:26:51.977173 8271 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:26:51.978782 8271 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:26:51.980007 8271 hybrid_clock.cc:648] HybridClock initialized: now 1786332411979954 us; error 37 us; skew 500 ppm
I20260810 03:26:51.980381 8271 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-08-10T03:26:52Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (2034 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786332336319544-28292-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 03:26:52 dist-test-slave-vpvm krb5kdc[8293](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 03:26:52 dist-test-slave-vpvm krb5kdc[8293](info): set up 2 sockets
Aug 10 03:26:52 dist-test-slave-vpvm krb5kdc[8293](info): commencing operation
krb5kdc: starting...
W20260810 03:26:53.306432 28292 mini_kdc.cc:121] Time spent starting KDC: real 1.201s user 0.006s sys 0.043s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786332336319544-28292-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786332336319544-28292-0/krb5kdc/test-user.keytab.
Aug 10 03:26:53 dist-test-slave-vpvm krb5kdc[8293](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332413, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T03:26:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T03:26:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.161.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.161.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.161.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.62.keytab.
Entry for principal kudu/127.27.161.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.62.keytab.
I20260810 03:26:53.606191 28292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iBH5/build/asan/bin/kudu
/tmp/dist-test-task24iBH5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786332336319544-28292-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786332336319544-28292-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786332336319544-28292-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786332336319544-28292-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.161.62:42221
--webserver_interface=127.27.161.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.62.keytab
--principal=kudu/127.27.161.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.161.20:36371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.161.62:42221
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786332336319544-28292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786332336319544-28292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786332336319544-28292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:26:53.959002 8314 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:26:53.959605 8314 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:26:53.959817 8314 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:26:53.969200 8314 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:26:53.969349 8314 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:26:53.969436 8314 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:26:53.969537 8314 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:26:53.969599 8314 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:26:53.982782 8314 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.161.20:36371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786332336319544-28292-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786332336319544-28292-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.161.62:42221
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.62.keytab
--principal=kudu/127.27.161.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.161.62:42221
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786332336319544-28292-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.161.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786332336319544-28292-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:26:53.984947 8314 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:26:53.986894 8314 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:26:54.000892 8320 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:26:54.000918 8321 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:26:54.002837 8323 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:26:54.003731 8314 server_base.cc:1034] running on GCE node
I20260810 03:26:54.004537 8314 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:26:54.006038 8314 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:26:54.007234 8314 hybrid_clock.cc:648] HybridClock initialized: now 1786332414007173 us; error 51 us; skew 500 ppm
I20260810 03:26:54.007587 8314 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:26:54 dist-test-slave-vpvm krb5kdc[8293](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332414, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.161.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:26:54.010671 8314 init.cc:377] Logged in from keytab as kudu/127.27.161.62@KRBTEST.COM (short username kudu)
I20260810 03:26:54.012212 8314 webserver.cc:460] Webserver started at http://127.27.161.62:35983/ using document root <none> and password file <none>
I20260810 03:26:54.012965 8314 fs_manager.cc:359] Metadata directory not provided
I20260810 03:26:54.013092 8314 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:26:54.013401 8314 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:26:54.016641 8314 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786332336319544-28292-0/minicluster-data/master-0/data/instance:
uuid: "ee63a7e951de4603bc5d1ab73dcd0910"
format_stamp: "Formatted at 2026-08-10 03:26:54 on dist-test-slave-vpvm"
server_key: "7990c5a1cc888c8635398e8be55969d8"
server_key_iv: "9c0cfc844636423675ab5ce880ff0eb0"
server_key_version: "encryptionkey@0"
I20260810 03:26:54.017449 8314 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786332336319544-28292-0/minicluster-data/master-0/wal/instance:
uuid: "ee63a7e951de4603bc5d1ab73dcd0910"
format_stamp: "Formatted at 2026-08-10 03:26:54 on dist-test-slave-vpvm"
server_key: "7990c5a1cc888c8635398e8be55969d8"
server_key_iv: "9c0cfc844636423675ab5ce880ff0eb0"
server_key_version: "encryptionkey@0"
I20260810 03:26:54.022933 8314 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260810 03:26:54.026494 8333 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:26:54.027817 8314 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260810 03:26:54.027987 8314 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786332336319544-28292-0/minicluster-data/master-0/data,/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786332336319544-28292-0/minicluster-data/master-0/wal
uuid: "ee63a7e951de4603bc5d1ab73dcd0910"
format_stamp: "Formatted at 2026-08-10 03:26:54 on dist-test-slave-vpvm"
server_key: "7990c5a1cc888c8635398e8be55969d8"
server_key_iv: "9c0cfc844636423675ab5ce880ff0eb0"
server_key_version: "encryptionkey@0"
I20260810 03:26:54.028189 8314 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786332336319544-28292-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786332336319544-28292-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786332336319544-28292-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:26:54.048111 8314 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:26:54.052373 8314 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:26:54.052649 8314 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:26:54.070163 8314 rpc_server.cc:307] RPC server started. Bound to: 127.27.161.62:42221
I20260810 03:26:54.070170 8415 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.161.62:42221 every 8 connection(s)
I20260810 03:26:54.071578 8314 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786332336319544-28292-0/minicluster-data/master-0/data/info.pb
I20260810 03:26:54.075424 8417 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:26:54.075865 28292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iBH5/build/asan/bin/kudu as pid 8314
I20260810 03:26:54.076121 28292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786332336319544-28292-0/minicluster-data/master-0/wal/instance
I20260810 03:26:54.076936 28292 external_mini_cluster.cc:1428] Setting key 53baef8be6a2a6ac1f13a4a1cf7343f2
I20260810 03:26:54.089852 8417 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee63a7e951de4603bc5d1ab73dcd0910: Bootstrap starting.
Aug 10 03:26:54 dist-test-slave-vpvm krb5kdc[8293](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332413, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.161.62@KRBTEST.COM
I20260810 03:26:54.093657 8417 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ee63a7e951de4603bc5d1ab73dcd0910: Neither blocks nor log segments found. Creating new log.
I20260810 03:26:54.095118 8417 log.cc:824] T 00000000000000000000000000000000 P ee63a7e951de4603bc5d1ab73dcd0910: Log is configured to *not* fsync() on all Append() calls
I20260810 03:26:54.098507 8417 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee63a7e951de4603bc5d1ab73dcd0910: No bootstrap required, opened a new log
I20260810 03:26:54.101248 8423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:54.079121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41300 (local address 127.27.161.62:42221)
0810 03:26:54.079946 (+ 825us) server_negotiation.cc:206] Beginning negotiation
0810 03:26:54.079961 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 03:26:54.080003 (+ 42us) server_negotiation.cc:407] Connection header received
0810 03:26:54.081888 (+ 1885us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:26:54.081912 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:26:54.082376 (+ 464us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:26:54.082732 (+ 356us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:26:54.084638 (+ 1906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:54.086013 (+ 1375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:54.087737 (+ 1724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:54.088404 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:54.088531 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:54.092973 (+ 4442us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:26:54.092985 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:26:54.093004 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:26:54.093053 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:26:54.096682 (+ 3629us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:26:54.097407 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:26:54.097416 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:26:54.097424 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:26:54.097633 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:26:54.097999 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:26:54.098004 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:26:54.098007 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:26:54.098740 (+ 733us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:26:54.099052 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0810 03:26:54.099395 (+ 343us) server_negotiation.cc:299] Negotiation successful
0810 03:26:54.099847 (+ 452us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":604,"thread_start_us":377,"threads_started":1}
I20260810 03:26:54.107435 8417 raft_consensus.cc:348] T 00000000000000000000000000000000 P ee63a7e951de4603bc5d1ab73dcd0910 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee63a7e951de4603bc5d1ab73dcd0910" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 42221 } }
I20260810 03:26:54.107784 8417 raft_consensus.cc:374] T 00000000000000000000000000000000 P ee63a7e951de4603bc5d1ab73dcd0910 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:26:54.107857 8417 raft_consensus.cc:729] T 00000000000000000000000000000000 P ee63a7e951de4603bc5d1ab73dcd0910 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee63a7e951de4603bc5d1ab73dcd0910, State: Initialized, Role: FOLLOWER
I20260810 03:26:54.108373 8417 consensus_queue.cc:260] T 00000000000000000000000000000000 P ee63a7e951de4603bc5d1ab73dcd0910 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee63a7e951de4603bc5d1ab73dcd0910" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 42221 } }
I20260810 03:26:54.108593 8417 raft_consensus.cc:388] T 00000000000000000000000000000000 P ee63a7e951de4603bc5d1ab73dcd0910 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 03:26:54.108702 8417 raft_consensus.cc:482] T 00000000000000000000000000000000 P ee63a7e951de4603bc5d1ab73dcd0910 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 03:26:54.108855 8417 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ee63a7e951de4603bc5d1ab73dcd0910 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:26:54.111209 8417 raft_consensus.cc:504] T 00000000000000000000000000000000 P ee63a7e951de4603bc5d1ab73dcd0910 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee63a7e951de4603bc5d1ab73dcd0910" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 42221 } }
I20260810 03:26:54.111703 8417 leader_election.cc:302] T 00000000000000000000000000000000 P ee63a7e951de4603bc5d1ab73dcd0910 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ee63a7e951de4603bc5d1ab73dcd0910; no voters:
I20260810 03:26:54.112287 8417 leader_election.cc:288] T 00000000000000000000000000000000 P ee63a7e951de4603bc5d1ab73dcd0910 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 03:26:54.112437 8425 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ee63a7e951de4603bc5d1ab73dcd0910 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:26:54.113469 8425 raft_consensus.cc:686] T 00000000000000000000000000000000 P ee63a7e951de4603bc5d1ab73dcd0910 [term 1 LEADER]: Becoming Leader. State: Replica: ee63a7e951de4603bc5d1ab73dcd0910, State: Running, Role: LEADER
I20260810 03:26:54.114046 8425 consensus_queue.cc:237] T 00000000000000000000000000000000 P ee63a7e951de4603bc5d1ab73dcd0910 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee63a7e951de4603bc5d1ab73dcd0910" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 42221 } }
I20260810 03:26:54.114555 8417 sys_catalog.cc:564] T 00000000000000000000000000000000 P ee63a7e951de4603bc5d1ab73dcd0910 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:26:54.117105 8428 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee63a7e951de4603bc5d1ab73dcd0910 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ee63a7e951de4603bc5d1ab73dcd0910. Latest consensus state: current_term: 1 leader_uuid: "ee63a7e951de4603bc5d1ab73dcd0910" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee63a7e951de4603bc5d1ab73dcd0910" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 42221 } } }
I20260810 03:26:54.117129 8427 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee63a7e951de4603bc5d1ab73dcd0910 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ee63a7e951de4603bc5d1ab73dcd0910" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee63a7e951de4603bc5d1ab73dcd0910" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 42221 } } }
I20260810 03:26:54.117537 8427 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee63a7e951de4603bc5d1ab73dcd0910 [sys.catalog]: This master's current role is: LEADER
I20260810 03:26:54.117537 8428 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee63a7e951de4603bc5d1ab73dcd0910 [sys.catalog]: This master's current role is: LEADER
I20260810 03:26:54.118989 8440 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 03:26:54.123728 8440 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 03:26:54.132776 8440 catalog_manager.cc:1347] Generated new cluster ID: 06317a3e03554fd6a9f71429c5c0ba12
I20260810 03:26:54.132890 8440 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 03:26:54.150765 8440 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 03:26:54.152626 8440 catalog_manager.cc:1504] Loading token signing keys...
I20260810 03:26:54.159516 8440 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ee63a7e951de4603bc5d1ab73dcd0910: Generated new TSK 0
I20260810 03:26:54.160519 8440 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.161.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.161.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.161.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.1.keytab.
Entry for principal kudu/127.27.161.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.1.keytab.
I20260810 03:26:54.237649 28292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iBH5/build/asan/bin/kudu
/tmp/dist-test-task24iBH5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786332336319544-28292-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786332336319544-28292-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786332336319544-28292-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786332336319544-28292-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.161.1:0
--local_ip_for_outbound_sockets=127.27.161.1
--webserver_interface=127.27.161.1
--webserver_port=0
--tserver_master_addrs=127.27.161.62:42221
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.1.keytab
--principal=kudu/127.27.161.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.161.20:36371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786332336319544-28292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786332336319544-28292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786332336319544-28292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:26:54.598543 8457 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:26:54.598829 8457 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:26:54.598932 8457 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:26:54.606429 8457 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:26:54.606552 8457 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:26:54.606673 8457 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.161.1
I20260810 03:26:54.616851 8457 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.161.20:36371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786332336319544-28292-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786332336319544-28292-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.1.keytab
--principal=kudu/127.27.161.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.161.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786332336319544-28292-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.161.1
--webserver_port=0
--tserver_master_addrs=127.27.161.62:42221
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.161.1
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786332336319544-28292-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:26:54.618227 8457 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:26:54.619590 8457 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:26:54.632608 8466 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:26:54.632591 8464 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:26:54.632797 8469 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:26:54.633612 8457 server_base.cc:1034] running on GCE node
I20260810 03:26:54.634245 8457 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:26:54.635164 8457 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:26:54.636363 8457 hybrid_clock.cc:648] HybridClock initialized: now 1786332414636313 us; error 43 us; skew 500 ppm
I20260810 03:26:54.636669 8457 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:26:54 dist-test-slave-vpvm krb5kdc[8293](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332414, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.161.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:26:54.639851 8457 init.cc:377] Logged in from keytab as kudu/127.27.161.1@KRBTEST.COM (short username kudu)
I20260810 03:26:54.641398 8457 webserver.cc:460] Webserver started at http://127.27.161.1:36345/ using document root <none> and password file <none>
I20260810 03:26:54.642171 8457 fs_manager.cc:359] Metadata directory not provided
I20260810 03:26:54.642295 8457 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:26:54.642604 8457 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:26:54.645617 8457 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786332336319544-28292-0/minicluster-data/ts-0/data/instance:
uuid: "a58b31fed7094c6da68ccd4da5e0058a"
format_stamp: "Formatted at 2026-08-10 03:26:54 on dist-test-slave-vpvm"
server_key: "cb46438e2c73ab8013c27e778a6f9e08"
server_key_iv: "8cb4e067705cc051f5296f79ba2daebd"
server_key_version: "encryptionkey@0"
I20260810 03:26:54.646476 8457 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786332336319544-28292-0/minicluster-data/ts-0/wal/instance:
uuid: "a58b31fed7094c6da68ccd4da5e0058a"
format_stamp: "Formatted at 2026-08-10 03:26:54 on dist-test-slave-vpvm"
server_key: "cb46438e2c73ab8013c27e778a6f9e08"
server_key_iv: "8cb4e067705cc051f5296f79ba2daebd"
server_key_version: "encryptionkey@0"
I20260810 03:26:54.651750 8457 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260810 03:26:54.655225 8481 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:26:54.656466 8457 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 03:26:54.656627 8457 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786332336319544-28292-0/minicluster-data/ts-0/data,/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786332336319544-28292-0/minicluster-data/ts-0/wal
uuid: "a58b31fed7094c6da68ccd4da5e0058a"
format_stamp: "Formatted at 2026-08-10 03:26:54 on dist-test-slave-vpvm"
server_key: "cb46438e2c73ab8013c27e778a6f9e08"
server_key_iv: "8cb4e067705cc051f5296f79ba2daebd"
server_key_version: "encryptionkey@0"
I20260810 03:26:54.656809 8457 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786332336319544-28292-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786332336319544-28292-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786332336319544-28292-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:26:54.680265 8457 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:26:54.684543 8457 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:26:54.684831 8457 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:26:54.685850 8457 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:26:54.687247 8457 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:26:54.687325 8457 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:26:54.687448 8457 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:26:54.687662 8457 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:26:54.720376 8457 rpc_server.cc:307] RPC server started. Bound to: 127.27.161.1:46369
I20260810 03:26:54.720427 8640 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.161.1:46369 every 8 connection(s)
I20260810 03:26:54.721882 8457 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786332336319544-28292-0/minicluster-data/ts-0/data/info.pb
I20260810 03:26:54.722402 28292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iBH5/build/asan/bin/kudu as pid 8457
I20260810 03:26:54.722654 28292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786332336319544-28292-0/minicluster-data/ts-0/wal/instance
I20260810 03:26:54.723464 28292 external_mini_cluster.cc:1428] Setting key e16c69a4065981aa39e8545da045b422
Aug 10 03:26:54 dist-test-slave-vpvm krb5kdc[8293](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332414, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.161.1@KRBTEST.COM for kudu/127.27.161.62@KRBTEST.COM
I20260810 03:26:54.743927 8643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:54.724678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.161.1:38113 (local address 127.27.161.62:42221)
0810 03:26:54.725164 (+ 486us) server_negotiation.cc:206] Beginning negotiation
0810 03:26:54.725171 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:26:54.726453 (+ 1282us) server_negotiation.cc:407] Connection header received
0810 03:26:54.728919 (+ 2466us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:26:54.728923 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:26:54.729000 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:26:54.729104 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:26:54.731144 (+ 2040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:54.731891 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:54.733995 (+ 2104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:54.734411 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:54.734512 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:54.738374 (+ 3862us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:26:54.738379 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:26:54.738382 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:26:54.738417 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:26:54.740776 (+ 2359us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:26:54.741537 (+ 761us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:26:54.741542 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:26:54.741544 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:26:54.741667 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:26:54.742132 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:26:54.742135 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:26:54.742138 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:26:54.742413 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:26:54.742562 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0810 03:26:54.743327 (+ 765us) server_negotiation.cc:299] Negotiation successful
0810 03:26:54.743572 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":356,"thread_start_us":238,"threads_started":1}
I20260810 03:26:54.744879 8644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:54.725394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:42221 (local address 127.27.161.1:38113)
0810 03:26:54.726280 (+ 886us) negotiation.cc:107] Waiting for socket to connect
0810 03:26:54.726334 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0810 03:26:54.728609 (+ 2275us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:26:54.729384 (+ 775us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:26:54.729394 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:26:54.730148 (+ 754us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:26:54.730936 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:26:54.730961 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:26:54.732052 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:26:54.732057 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:26:54.733836 (+ 1779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:26:54.733851 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:26:54.734641 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:26:54.734648 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:26:54.734919 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:54.735633 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:26:54.735662 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:26:54.738159 (+ 2497us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:26:54.740915 (+ 2756us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:26:54.740925 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:26:54.740943 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:26:54.741403 (+ 460us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:26:54.741775 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:26:54.741780 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:26:54.741784 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:26:54.742011 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:26:54.742552 (+ 541us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:26:54.742560 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:26:54.742901 (+ 341us) client_negotiation.cc:769] Sending connection context
0810 03:26:54.743162 (+ 261us) client_negotiation.cc:240] Negotiation successful
0810 03:26:54.743711 (+ 549us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":575,"thread_start_us":314,"threads_started":1}
I20260810 03:26:54.746441 8641 heartbeater.cc:347] Connected to a master server at 127.27.161.62:42221
I20260810 03:26:54.746754 8641 heartbeater.cc:464] Registering TS with master...
I20260810 03:26:54.747471 8641 heartbeater.cc:511] Master 127.27.161.62:42221 requested a full tablet report, sending...
I20260810 03:26:54.749547 8361 ts_manager.cc:194] Registered new tserver with Master: a58b31fed7094c6da68ccd4da5e0058a (127.27.161.1:46369)
I20260810 03:26:54.751072 8361 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.161.1@KRBTEST.COM'} at 127.27.161.1:38113
WARNING: no policy specified for kudu/127.27.161.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.161.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.161.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.2.keytab.
Entry for principal kudu/127.27.161.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.2.keytab.
I20260810 03:26:54.787133 28292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iBH5/build/asan/bin/kudu
/tmp/dist-test-task24iBH5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786332336319544-28292-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786332336319544-28292-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786332336319544-28292-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786332336319544-28292-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.161.2:0
--local_ip_for_outbound_sockets=127.27.161.2
--webserver_interface=127.27.161.2
--webserver_port=0
--tserver_master_addrs=127.27.161.62:42221
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.2.keytab
--principal=kudu/127.27.161.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.161.20:36371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786332336319544-28292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786332336319544-28292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786332336319544-28292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:26:55.141881 8649 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:26:55.142201 8649 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:26:55.142304 8649 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:26:55.150075 8649 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:26:55.150238 8649 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:26:55.150362 8649 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.161.2
I20260810 03:26:55.161087 8649 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.161.20:36371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786332336319544-28292-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786332336319544-28292-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.2.keytab
--principal=kudu/127.27.161.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.161.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786332336319544-28292-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.161.2
--webserver_port=0
--tserver_master_addrs=127.27.161.62:42221
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.161.2
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786332336319544-28292-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:26:55.162529 8649 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:26:55.164021 8649 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:26:55.177531 8661 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:26:55.177603 8663 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:26:55.177512 8658 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:26:55.178360 8649 server_base.cc:1034] running on GCE node
I20260810 03:26:55.179026 8649 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:26:55.180032 8649 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:26:55.181241 8649 hybrid_clock.cc:648] HybridClock initialized: now 1786332415181188 us; error 38 us; skew 500 ppm
I20260810 03:26:55.181568 8649 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:26:55 dist-test-slave-vpvm krb5kdc[8293](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332415, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.161.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:26:55.184787 8649 init.cc:377] Logged in from keytab as kudu/127.27.161.2@KRBTEST.COM (short username kudu)
I20260810 03:26:55.186398 8649 webserver.cc:460] Webserver started at http://127.27.161.2:34099/ using document root <none> and password file <none>
I20260810 03:26:55.187194 8649 fs_manager.cc:359] Metadata directory not provided
I20260810 03:26:55.187314 8649 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:26:55.187630 8649 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:26:55.190721 8649 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786332336319544-28292-0/minicluster-data/ts-1/data/instance:
uuid: "2b27714d2680479984776b4b44aff0d9"
format_stamp: "Formatted at 2026-08-10 03:26:55 on dist-test-slave-vpvm"
server_key: "ae6d67ea236403936aeb53943b36d0d8"
server_key_iv: "e43241d3e2e0fb5694481bfced0635d4"
server_key_version: "encryptionkey@0"
I20260810 03:26:55.191530 8649 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786332336319544-28292-0/minicluster-data/ts-1/wal/instance:
uuid: "2b27714d2680479984776b4b44aff0d9"
format_stamp: "Formatted at 2026-08-10 03:26:55 on dist-test-slave-vpvm"
server_key: "ae6d67ea236403936aeb53943b36d0d8"
server_key_iv: "e43241d3e2e0fb5694481bfced0635d4"
server_key_version: "encryptionkey@0"
I20260810 03:26:55.197191 8649 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260810 03:26:55.201526 8675 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:26:55.203032 8649 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 03:26:55.203207 8649 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786332336319544-28292-0/minicluster-data/ts-1/data,/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786332336319544-28292-0/minicluster-data/ts-1/wal
uuid: "2b27714d2680479984776b4b44aff0d9"
format_stamp: "Formatted at 2026-08-10 03:26:55 on dist-test-slave-vpvm"
server_key: "ae6d67ea236403936aeb53943b36d0d8"
server_key_iv: "e43241d3e2e0fb5694481bfced0635d4"
server_key_version: "encryptionkey@0"
I20260810 03:26:55.203428 8649 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786332336319544-28292-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786332336319544-28292-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786332336319544-28292-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:26:55.222234 8649 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:26:55.226256 8649 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:26:55.226536 8649 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:26:55.227558 8649 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:26:55.228977 8649 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:26:55.229053 8649 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:26:55.229185 8649 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:26:55.229236 8649 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:26:55.260617 8649 rpc_server.cc:307] RPC server started. Bound to: 127.27.161.2:39655
I20260810 03:26:55.260710 8840 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.161.2:39655 every 8 connection(s)
I20260810 03:26:55.261942 8649 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786332336319544-28292-0/minicluster-data/ts-1/data/info.pb
I20260810 03:26:55.268525 28292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iBH5/build/asan/bin/kudu as pid 8649
I20260810 03:26:55.268705 28292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786332336319544-28292-0/minicluster-data/ts-1/wal/instance
I20260810 03:26:55.269255 28292 external_mini_cluster.cc:1428] Setting key 84474dc0094e29b940c179be111cfaf2
Aug 10 03:26:55 dist-test-slave-vpvm krb5kdc[8293](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332415, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.161.2@KRBTEST.COM for kudu/127.27.161.62@KRBTEST.COM
I20260810 03:26:55.281632 8845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:55.264562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.161.2:39403 (local address 127.27.161.62:42221)
0810 03:26:55.264980 (+ 418us) server_negotiation.cc:206] Beginning negotiation
0810 03:26:55.264985 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:26:55.265903 (+ 918us) server_negotiation.cc:407] Connection header received
0810 03:26:55.267758 (+ 1855us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:26:55.267764 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:26:55.267843 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:26:55.267940 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:26:55.269633 (+ 1693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:55.270500 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:55.272191 (+ 1691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:55.272615 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:55.272728 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:55.276118 (+ 3390us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:26:55.276123 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:26:55.276127 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:26:55.276162 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:26:55.278638 (+ 2476us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:26:55.279351 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:26:55.279357 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:26:55.279360 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:26:55.279486 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:26:55.279903 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:26:55.279908 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:26:55.279911 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:26:55.280181 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:26:55.280317 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0810 03:26:55.281010 (+ 693us) server_negotiation.cc:299] Negotiation successful
0810 03:26:55.281244 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":318,"thread_start_us":217,"threads_started":1}
I20260810 03:26:55.282680 8846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:55.265019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:42221 (local address 127.27.161.2:39403)
0810 03:26:55.265742 (+ 723us) negotiation.cc:107] Waiting for socket to connect
0810 03:26:55.265790 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0810 03:26:55.267483 (+ 1693us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:26:55.268215 (+ 732us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:26:55.268224 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:26:55.268793 (+ 569us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:26:55.269454 (+ 661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:26:55.269475 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:26:55.270674 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:26:55.270678 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:26:55.272038 (+ 1360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:26:55.272052 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:26:55.272852 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:26:55.272861 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:26:55.273069 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:55.273702 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:26:55.273730 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:26:55.275885 (+ 2155us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:26:55.278777 (+ 2892us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:26:55.278786 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:26:55.278804 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:26:55.279233 (+ 429us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:26:55.279596 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:26:55.279601 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:26:55.279605 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:26:55.279794 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:26:55.280310 (+ 516us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:26:55.280318 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:26:55.280654 (+ 336us) client_negotiation.cc:769] Sending connection context
0810 03:26:55.280905 (+ 251us) client_negotiation.cc:240] Negotiation successful
0810 03:26:55.281463 (+ 558us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":281,"threads_started":1}
I20260810 03:26:55.284159 8843 heartbeater.cc:347] Connected to a master server at 127.27.161.62:42221
I20260810 03:26:55.284456 8843 heartbeater.cc:464] Registering TS with master...
I20260810 03:26:55.285262 8843 heartbeater.cc:511] Master 127.27.161.62:42221 requested a full tablet report, sending...
I20260810 03:26:55.286973 8361 ts_manager.cc:194] Registered new tserver with Master: 2b27714d2680479984776b4b44aff0d9 (127.27.161.2:39655)
I20260810 03:26:55.288141 8361 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.161.2@KRBTEST.COM'} at 127.27.161.2:39403
WARNING: no policy specified for kudu/127.27.161.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.161.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.161.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.3.keytab.
Entry for principal kudu/127.27.161.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.3.keytab.
I20260810 03:26:55.331316 28292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iBH5/build/asan/bin/kudu
/tmp/dist-test-task24iBH5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786332336319544-28292-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786332336319544-28292-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786332336319544-28292-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786332336319544-28292-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.161.3:0
--local_ip_for_outbound_sockets=127.27.161.3
--webserver_interface=127.27.161.3
--webserver_port=0
--tserver_master_addrs=127.27.161.62:42221
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.3.keytab
--principal=kudu/127.27.161.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.161.20:36371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786332336319544-28292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786332336319544-28292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786332336319544-28292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:26:55.682678 8850 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:26:55.682983 8850 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:26:55.683077 8850 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:26:55.690362 8850 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:26:55.690482 8850 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:26:55.690601 8850 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.161.3
I20260810 03:26:55.700444 8850 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.161.20:36371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786332336319544-28292-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786332336319544-28292-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.3.keytab
--principal=kudu/127.27.161.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.161.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786332336319544-28292-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.161.3
--webserver_port=0
--tserver_master_addrs=127.27.161.62:42221
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.161.3
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786332336319544-28292-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:26:55.701843 8850 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:26:55.703208 8850 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:26:55.716845 8862 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:26:55.716814 8868 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:26:55.716876 8864 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:26:55.717617 8850 server_base.cc:1034] running on GCE node
I20260810 03:26:55.718371 8850 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:26:55.719420 8850 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:26:55.720635 8850 hybrid_clock.cc:648] HybridClock initialized: now 1786332415720573 us; error 53 us; skew 500 ppm
I20260810 03:26:55.720958 8850 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:26:55 dist-test-slave-vpvm krb5kdc[8293](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332415, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.161.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:26:55.724355 8850 init.cc:377] Logged in from keytab as kudu/127.27.161.3@KRBTEST.COM (short username kudu)
I20260810 03:26:55.725914 8850 webserver.cc:460] Webserver started at http://127.27.161.3:41747/ using document root <none> and password file <none>
I20260810 03:26:55.726626 8850 fs_manager.cc:359] Metadata directory not provided
I20260810 03:26:55.726732 8850 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:26:55.727005 8850 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:26:55.730046 8850 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786332336319544-28292-0/minicluster-data/ts-2/data/instance:
uuid: "8a828ad0fd024a97aec877395b55b090"
format_stamp: "Formatted at 2026-08-10 03:26:55 on dist-test-slave-vpvm"
server_key: "6ae662c75abdc5215012053e41537e52"
server_key_iv: "dac77c5099396acfcf409c80b7dc920d"
server_key_version: "encryptionkey@0"
I20260810 03:26:55.730850 8850 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786332336319544-28292-0/minicluster-data/ts-2/wal/instance:
uuid: "8a828ad0fd024a97aec877395b55b090"
format_stamp: "Formatted at 2026-08-10 03:26:55 on dist-test-slave-vpvm"
server_key: "6ae662c75abdc5215012053e41537e52"
server_key_iv: "dac77c5099396acfcf409c80b7dc920d"
server_key_version: "encryptionkey@0"
I20260810 03:26:55.736263 8850 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260810 03:26:55.739904 8882 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:26:55.741263 8850 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 03:26:55.741415 8850 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786332336319544-28292-0/minicluster-data/ts-2/data,/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786332336319544-28292-0/minicluster-data/ts-2/wal
uuid: "8a828ad0fd024a97aec877395b55b090"
format_stamp: "Formatted at 2026-08-10 03:26:55 on dist-test-slave-vpvm"
server_key: "6ae662c75abdc5215012053e41537e52"
server_key_iv: "dac77c5099396acfcf409c80b7dc920d"
server_key_version: "encryptionkey@0"
I20260810 03:26:55.741592 8850 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786332336319544-28292-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786332336319544-28292-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786332336319544-28292-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:26:55.754179 8641 heartbeater.cc:503] Master 127.27.161.62:42221 was elected leader, sending a full tablet report...
I20260810 03:26:55.771499 8850 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:26:55.775965 8850 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:26:55.776281 8850 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:26:55.777341 8850 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:26:55.778864 8850 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:26:55.778940 8850 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:26:55.779055 8850 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:26:55.779121 8850 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:26:55.809298 8850 rpc_server.cc:307] RPC server started. Bound to: 127.27.161.3:35967
I20260810 03:26:55.809345 9047 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.161.3:35967 every 8 connection(s)
I20260810 03:26:55.810599 8850 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786332336319544-28292-0/minicluster-data/ts-2/data/info.pb
I20260810 03:26:55.811400 28292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iBH5/build/asan/bin/kudu as pid 8850
I20260810 03:26:55.811600 28292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786332336319544-28292-0/minicluster-data/ts-2/wal/instance
I20260810 03:26:55.812170 28292 external_mini_cluster.cc:1428] Setting key 40cc48ed7097ef0b7a382f146b795478
Aug 10 03:26:55 dist-test-slave-vpvm krb5kdc[8293](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332415, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.161.3@KRBTEST.COM for kudu/127.27.161.62@KRBTEST.COM
I20260810 03:26:55.831288 9050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:55.813627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.161.3:53383 (local address 127.27.161.62:42221)
0810 03:26:55.814147 (+ 520us) server_negotiation.cc:206] Beginning negotiation
0810 03:26:55.814152 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:26:55.815359 (+ 1207us) server_negotiation.cc:407] Connection header received
0810 03:26:55.817280 (+ 1921us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:26:55.817284 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:26:55.817365 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:26:55.817453 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:26:55.819065 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:55.819928 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:55.821411 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:55.821810 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:55.821897 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:55.825264 (+ 3367us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:26:55.825269 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:26:55.825272 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:26:55.825307 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:26:55.828330 (+ 3023us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:26:55.829087 (+ 757us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:26:55.829091 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:26:55.829093 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:26:55.829217 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:26:55.829631 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:26:55.829633 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:26:55.829635 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:26:55.829911 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:26:55.830086 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0810 03:26:55.830763 (+ 677us) server_negotiation.cc:299] Negotiation successful
0810 03:26:55.830961 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":395,"thread_start_us":329,"threads_started":1}
I20260810 03:26:55.832167 9051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:55.814385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:42221 (local address 127.27.161.3:53383)
0810 03:26:55.815194 (+ 809us) negotiation.cc:107] Waiting for socket to connect
0810 03:26:55.815245 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0810 03:26:55.817025 (+ 1780us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:26:55.817662 (+ 637us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:26:55.817670 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:26:55.818257 (+ 587us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:26:55.818922 (+ 665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:26:55.818942 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:26:55.820079 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:26:55.820083 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:26:55.821301 (+ 1218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:26:55.821313 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:26:55.822028 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:26:55.822035 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:26:55.822245 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:55.822847 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:26:55.822870 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:26:55.825049 (+ 2179us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:26:55.828478 (+ 3429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:26:55.828488 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:26:55.828505 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:26:55.828982 (+ 477us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:26:55.829316 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:26:55.829320 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:26:55.829323 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:26:55.829544 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:26:55.830065 (+ 521us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:26:55.830071 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:26:55.830423 (+ 352us) client_negotiation.cc:769] Sending connection context
0810 03:26:55.830651 (+ 228us) client_negotiation.cc:240] Negotiation successful
0810 03:26:55.831051 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":507,"thread_start_us":287,"threads_started":1}
I20260810 03:26:55.833542 9048 heartbeater.cc:347] Connected to a master server at 127.27.161.62:42221
I20260810 03:26:55.833832 9048 heartbeater.cc:464] Registering TS with master...
I20260810 03:26:55.834589 9048 heartbeater.cc:511] Master 127.27.161.62:42221 requested a full tablet report, sending...
I20260810 03:26:55.835984 8361 ts_manager.cc:194] Registered new tserver with Master: 8a828ad0fd024a97aec877395b55b090 (127.27.161.3:35967)
I20260810 03:26:55.836752 8361 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.161.3@KRBTEST.COM'} at 127.27.161.3:53383
I20260810 03:26:55.840930 28292 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.27.161.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.161.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.161.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786332336319544-28292-0/krb5kdc/oryx_127.27.161.1.keytab.
Entry for principal oryx/127.27.161.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786332336319544-28292-0/krb5kdc/oryx_127.27.161.1.keytab.
WARNING: no policy specified for oryx/127.27.161.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.161.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.161.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786332336319544-28292-0/krb5kdc/oryx_127.27.161.2.keytab.
Entry for principal oryx/127.27.161.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786332336319544-28292-0/krb5kdc/oryx_127.27.161.2.keytab.
WARNING: no policy specified for oryx/127.27.161.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.161.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.161.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786332336319544-28292-0/krb5kdc/oryx_127.27.161.3.keytab.
Entry for principal oryx/127.27.161.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786332336319544-28292-0/krb5kdc/oryx_127.27.161.3.keytab.
I20260810 03:26:56.020813 28292 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iBH5/build/asan/bin/kudu pid 8457
Aug 10 03:26:56 dist-test-slave-vpvm krb5kdc[8293](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332413, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.161.1@KRBTEST.COM
I20260810 03:26:56.037794 9064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:56.021974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45962 (local address 127.27.161.1:46369)
0810 03:26:56.022854 (+ 880us) server_negotiation.cc:206] Beginning negotiation
0810 03:26:56.022861 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:26:56.022890 (+ 29us) server_negotiation.cc:407] Connection header received
0810 03:26:56.022954 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:26:56.022960 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:26:56.023185 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:26:56.023333 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:26:56.024731 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:56.025784 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:56.027029 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:56.027621 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:56.027721 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:56.032015 (+ 4294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:26:56.032038 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:26:56.032060 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:26:56.032105 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:26:56.035064 (+ 2959us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:26:56.035861 (+ 797us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:26:56.035868 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:26:56.035870 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:26:56.035997 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:26:56.036312 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:26:56.036315 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:26:56.036317 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:26:56.036688 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:26:56.036930 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0810 03:26:56.037204 (+ 274us) server_negotiation.cc:299] Negotiation successful
0810 03:26:56.037414 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":730,"thread_start_us":529,"threads_started":1}
I20260810 03:26:56.039111 8635 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:26:56.123023 28292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iBH5/build/asan/bin/kudu with pid 8457
I20260810 03:26:56.148240 28292 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iBH5/build/asan/bin/kudu pid 8649
Aug 10 03:26:56 dist-test-slave-vpvm krb5kdc[8293](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332413, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.161.2@KRBTEST.COM
I20260810 03:26:56.165429 9067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:56.149146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52134 (local address 127.27.161.2:39655)
0810 03:26:56.149686 (+ 540us) server_negotiation.cc:206] Beginning negotiation
0810 03:26:56.149694 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 03:26:56.149721 (+ 27us) server_negotiation.cc:407] Connection header received
0810 03:26:56.149789 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:26:56.149796 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:26:56.150049 (+ 253us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:26:56.150183 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:26:56.151373 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:56.152385 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:56.153812 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:56.154682 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:56.154808 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:56.159321 (+ 4513us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:26:56.159339 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:26:56.159358 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:26:56.159404 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:26:56.162491 (+ 3087us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:26:56.163258 (+ 767us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:26:56.163267 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:26:56.163269 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:26:56.163412 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:26:56.163850 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:26:56.163854 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:26:56.163856 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:26:56.164271 (+ 415us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:26:56.164515 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0810 03:26:56.164821 (+ 306us) server_negotiation.cc:299] Negotiation successful
0810 03:26:56.165043 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":389,"thread_start_us":240,"threads_started":1}
I20260810 03:26:56.166846 8833 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:26:56.250841 28292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iBH5/build/asan/bin/kudu with pid 8649
I20260810 03:26:56.284173 28292 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iBH5/build/asan/bin/kudu pid 8850
Aug 10 03:26:56 dist-test-slave-vpvm krb5kdc[8293](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332413, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.161.3@KRBTEST.COM
I20260810 03:26:56.301265 9072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:56.285306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60894 (local address 127.27.161.3:35967)
0810 03:26:56.285990 (+ 684us) server_negotiation.cc:206] Beginning negotiation
0810 03:26:56.286000 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 03:26:56.286041 (+ 41us) server_negotiation.cc:407] Connection header received
0810 03:26:56.286124 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:26:56.286134 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:26:56.286427 (+ 293us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:26:56.286611 (+ 184us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:26:56.287957 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:56.288931 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:56.290287 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:56.290859 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:56.290978 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:56.295332 (+ 4354us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:26:56.295346 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:26:56.295363 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:26:56.295402 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:26:56.298395 (+ 2993us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:26:56.299109 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:26:56.299118 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:26:56.299120 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:26:56.299252 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:26:56.299637 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:26:56.299641 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:26:56.299643 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:26:56.300011 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:26:56.300317 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0810 03:26:56.300646 (+ 329us) server_negotiation.cc:299] Negotiation successful
0810 03:26:56.300923 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":482,"thread_start_us":282,"threads_started":1}
I20260810 03:26:56.302683 9042 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:26:56.386106 28292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iBH5/build/asan/bin/kudu with pid 8850
I20260810 03:26:56.409778 28292 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iBH5/build/asan/bin/kudu pid 8314
I20260810 03:26:56.411027 8408 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:26:56.515858 28292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iBH5/build/asan/bin/kudu with pid 8314
I20260810 03:26:56.547020 28292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iBH5/build/asan/bin/kudu
/tmp/dist-test-task24iBH5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786332336319544-28292-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786332336319544-28292-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786332336319544-28292-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786332336319544-28292-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.161.62:42221
--webserver_interface=127.27.161.62
--webserver_port=35983
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.62.keytab
--principal=kudu/127.27.161.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.161.20:36371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.161.62:42221
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786332336319544-28292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786332336319544-28292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786332336319544-28292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:26:56.893905 9075 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:26:56.894289 9075 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:26:56.894435 9075 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:26:56.901779 9075 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:26:56.901917 9075 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:26:56.902022 9075 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:26:56.902104 9075 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:26:56.902168 9075 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:26:56.912390 9075 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.161.20:36371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786332336319544-28292-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786332336319544-28292-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.161.62:42221
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.62.keytab
--principal=kudu/127.27.161.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.161.62:42221
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786332336319544-28292-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.161.62
--webserver_port=35983
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786332336319544-28292-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:26:56.913825 9075 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:26:56.915234 9075 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:26:56.927767 9082 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:26:56.927805 9086 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:26:56.927688 9084 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:26:56.928907 9075 server_base.cc:1034] running on GCE node
I20260810 03:26:56.929816 9075 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:26:56.931461 9075 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:26:56.932714 9075 hybrid_clock.cc:648] HybridClock initialized: now 1786332416932647 us; error 57 us; skew 500 ppm
I20260810 03:26:56.933104 9075 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:26:56 dist-test-slave-vpvm krb5kdc[8293](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332416, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.161.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:26:56.936412 9075 init.cc:377] Logged in from keytab as kudu/127.27.161.62@KRBTEST.COM (short username kudu)
I20260810 03:26:56.938033 9075 webserver.cc:460] Webserver started at http://127.27.161.62:35983/ using document root <none> and password file <none>
I20260810 03:26:56.938808 9075 fs_manager.cc:359] Metadata directory not provided
I20260810 03:26:56.938937 9075 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:26:56.945611 9075 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260810 03:26:56.948993 9096 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:26:56.950766 9075 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260810 03:26:56.950969 9075 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786332336319544-28292-0/minicluster-data/master-0/data,/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786332336319544-28292-0/minicluster-data/master-0/wal
uuid: "ee63a7e951de4603bc5d1ab73dcd0910"
format_stamp: "Formatted at 2026-08-10 03:26:54 on dist-test-slave-vpvm"
server_key: "7990c5a1cc888c8635398e8be55969d8"
server_key_iv: "9c0cfc844636423675ab5ce880ff0eb0"
server_key_version: "encryptionkey@0"
I20260810 03:26:56.952039 9075 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786332336319544-28292-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786332336319544-28292-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786332336319544-28292-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:26:56.969498 9075 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:26:56.974161 9075 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:26:56.974457 9075 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:26:56.993305 9075 rpc_server.cc:307] RPC server started. Bound to: 127.27.161.62:42221
I20260810 03:26:56.993319 9169 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.161.62:42221 every 8 connection(s)
I20260810 03:26:56.994987 9075 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786332336319544-28292-0/minicluster-data/master-0/data/info.pb
I20260810 03:26:56.996860 28292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iBH5/build/asan/bin/kudu as pid 9075
I20260810 03:26:56.997668 28292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iBH5/build/asan/bin/kudu
/tmp/dist-test-task24iBH5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786332336319544-28292-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786332336319544-28292-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786332336319544-28292-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786332336319544-28292-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.161.1:46369
--local_ip_for_outbound_sockets=127.27.161.1
--tserver_master_addrs=127.27.161.62:42221
--webserver_port=36345
--webserver_interface=127.27.161.1
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.1.keytab
--principal=kudu/127.27.161.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.161.20:36371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.27.161.1
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786332336319544-28292-0/krb5kdc/oryx_127.27.161.1.keytab with env {KRB5CCNAME=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786332336319544-28292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786332336319544-28292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786332336319544-28292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 03:26:57.002470 9170 sys_catalog.cc:263] Verifying existing consensus state
I20260810 03:26:57.006492 9170 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee63a7e951de4603bc5d1ab73dcd0910: Bootstrap starting.
I20260810 03:26:57.032545 9170 log.cc:824] T 00000000000000000000000000000000 P ee63a7e951de4603bc5d1ab73dcd0910: Log is configured to *not* fsync() on all Append() calls
I20260810 03:26:57.044713 9170 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee63a7e951de4603bc5d1ab73dcd0910: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 03:26:57.045375 9170 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee63a7e951de4603bc5d1ab73dcd0910: Bootstrap complete.
I20260810 03:26:57.056804 9170 raft_consensus.cc:348] T 00000000000000000000000000000000 P ee63a7e951de4603bc5d1ab73dcd0910 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee63a7e951de4603bc5d1ab73dcd0910" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 42221 } }
I20260810 03:26:57.057770 9170 raft_consensus.cc:729] T 00000000000000000000000000000000 P ee63a7e951de4603bc5d1ab73dcd0910 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee63a7e951de4603bc5d1ab73dcd0910, State: Initialized, Role: FOLLOWER
I20260810 03:26:57.058511 9170 consensus_queue.cc:260] T 00000000000000000000000000000000 P ee63a7e951de4603bc5d1ab73dcd0910 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee63a7e951de4603bc5d1ab73dcd0910" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 42221 } }
I20260810 03:26:57.058777 9170 raft_consensus.cc:388] T 00000000000000000000000000000000 P ee63a7e951de4603bc5d1ab73dcd0910 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 03:26:57.058910 9170 raft_consensus.cc:482] T 00000000000000000000000000000000 P ee63a7e951de4603bc5d1ab73dcd0910 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 03:26:57.059119 9170 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ee63a7e951de4603bc5d1ab73dcd0910 [term 1 FOLLOWER]: Advancing to term 2
I20260810 03:26:57.063380 9170 raft_consensus.cc:504] T 00000000000000000000000000000000 P ee63a7e951de4603bc5d1ab73dcd0910 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee63a7e951de4603bc5d1ab73dcd0910" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 42221 } }
I20260810 03:26:57.063939 9170 leader_election.cc:302] T 00000000000000000000000000000000 P ee63a7e951de4603bc5d1ab73dcd0910 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ee63a7e951de4603bc5d1ab73dcd0910; no voters:
I20260810 03:26:57.064579 9170 leader_election.cc:288] T 00000000000000000000000000000000 P ee63a7e951de4603bc5d1ab73dcd0910 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260810 03:26:57.064826 9177 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ee63a7e951de4603bc5d1ab73dcd0910 [term 2 FOLLOWER]: Leader election won for term 2
I20260810 03:26:57.066133 9177 raft_consensus.cc:686] T 00000000000000000000000000000000 P ee63a7e951de4603bc5d1ab73dcd0910 [term 2 LEADER]: Becoming Leader. State: Replica: ee63a7e951de4603bc5d1ab73dcd0910, State: Running, Role: LEADER
I20260810 03:26:57.066929 9170 sys_catalog.cc:564] T 00000000000000000000000000000000 P ee63a7e951de4603bc5d1ab73dcd0910 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:26:57.066871 9177 consensus_queue.cc:237] T 00000000000000000000000000000000 P ee63a7e951de4603bc5d1ab73dcd0910 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee63a7e951de4603bc5d1ab73dcd0910" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 42221 } }
I20260810 03:26:57.069787 9180 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee63a7e951de4603bc5d1ab73dcd0910 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ee63a7e951de4603bc5d1ab73dcd0910. Latest consensus state: current_term: 2 leader_uuid: "ee63a7e951de4603bc5d1ab73dcd0910" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee63a7e951de4603bc5d1ab73dcd0910" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 42221 } } }
I20260810 03:26:57.070230 9180 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee63a7e951de4603bc5d1ab73dcd0910 [sys.catalog]: This master's current role is: LEADER
I20260810 03:26:57.070782 9178 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee63a7e951de4603bc5d1ab73dcd0910 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "ee63a7e951de4603bc5d1ab73dcd0910" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee63a7e951de4603bc5d1ab73dcd0910" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 42221 } } }
I20260810 03:26:57.071108 9178 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee63a7e951de4603bc5d1ab73dcd0910 [sys.catalog]: This master's current role is: LEADER
I20260810 03:26:57.079303 9188 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 03:26:57.087008 9188 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 03:26:57.088515 9188 catalog_manager.cc:1259] Loaded cluster ID: 06317a3e03554fd6a9f71429c5c0ba12
I20260810 03:26:57.088634 9188 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 03:26:57.092641 9188 catalog_manager.cc:1504] Loading token signing keys...
I20260810 03:26:57.094179 9188 catalog_manager.cc:5879] T 00000000000000000000000000000000 P ee63a7e951de4603bc5d1ab73dcd0910: Loaded TSK: 0
I20260810 03:26:57.095791 9188 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 03:26:57.097811 9201 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
W20260810 03:26:57.377830 9174 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:26:57.378187 9174 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:26:57.378334 9174 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:26:57.386379 9174 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:26:57.386516 9174 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:26:57.386672 9174 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.161.1
I20260810 03:26:57.397244 9174 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.161.20:36371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786332336319544-28292-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786332336319544-28292-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786332336319544-28292-0/krb5kdc/oryx_127.27.161.1.keytab
--principal=oryx/127.27.161.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.161.1:46369
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786332336319544-28292-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.161.1
--webserver_port=36345
--tserver_master_addrs=127.27.161.62:42221
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.161.1
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786332336319544-28292-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:26:57.398806 9174 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:26:57.400256 9174 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:26:57.415683 9210 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:26:57.415678 9211 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:26:57.416204 9213 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:26:57.417412 9174 server_base.cc:1034] running on GCE node
I20260810 03:26:57.418182 9174 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:26:57.419353 9174 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:26:57.420603 9174 hybrid_clock.cc:648] HybridClock initialized: now 1786332417420531 us; error 63 us; skew 500 ppm
I20260810 03:26:57.420991 9174 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:26:57 dist-test-slave-vpvm krb5kdc[8293](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332417, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.161.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:26:57.428783 9174 init.cc:377] Logged in from keytab as oryx/127.27.161.1@KRBTEST.COM (short username oryx)
I20260810 03:26:57.430667 9174 webserver.cc:460] Webserver started at http://127.27.161.1:36345/ using document root <none> and password file <none>
I20260810 03:26:57.431653 9174 fs_manager.cc:359] Metadata directory not provided
I20260810 03:26:57.431957 9174 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:26:57.438158 9174 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260810 03:26:57.440979 9226 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:26:57.442420 9174 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 03:26:57.442584 9174 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786332336319544-28292-0/minicluster-data/ts-0/data,/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786332336319544-28292-0/minicluster-data/ts-0/wal
uuid: "a58b31fed7094c6da68ccd4da5e0058a"
format_stamp: "Formatted at 2026-08-10 03:26:54 on dist-test-slave-vpvm"
server_key: "cb46438e2c73ab8013c27e778a6f9e08"
server_key_iv: "8cb4e067705cc051f5296f79ba2daebd"
server_key_version: "encryptionkey@0"
I20260810 03:26:57.443574 9174 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786332336319544-28292-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786332336319544-28292-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786332336319544-28292-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:26:57.456229 9174 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:26:57.461092 9174 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:26:57.461367 9174 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:26:57.462437 9174 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:26:57.463903 9174 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:26:57.464016 9174 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:26:57.464130 9174 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:26:57.464211 9174 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:26:57.498823 9174 rpc_server.cc:307] RPC server started. Bound to: 127.27.161.1:46369
I20260810 03:26:57.498893 9388 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.161.1:46369 every 8 connection(s)
I20260810 03:26:57.500339 9174 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786332336319544-28292-0/minicluster-data/ts-0/data/info.pb
I20260810 03:26:57.505879 28292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iBH5/build/asan/bin/kudu as pid 9174
I20260810 03:26:57.506676 28292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iBH5/build/asan/bin/kudu
/tmp/dist-test-task24iBH5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786332336319544-28292-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786332336319544-28292-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786332336319544-28292-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786332336319544-28292-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.161.2:39655
--local_ip_for_outbound_sockets=127.27.161.2
--tserver_master_addrs=127.27.161.62:42221
--webserver_port=34099
--webserver_interface=127.27.161.2
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.2.keytab
--principal=kudu/127.27.161.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.161.20:36371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.27.161.2
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786332336319544-28292-0/krb5kdc/oryx_127.27.161.2.keytab with env {KRB5CCNAME=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786332336319544-28292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786332336319544-28292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786332336319544-28292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 03:26:57 dist-test-slave-vpvm krb5kdc[8293](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.161.1@KRBTEST.COM for oryx/127.27.161.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 03:26:57 dist-test-slave-vpvm krb5kdc[8293](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.161.1@KRBTEST.COM for oryx/127.27.161.62@KRBTEST.COM, Server not found in Kerberos database
W20260810 03:26:57.858722 9395 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:26:57.859033 9395 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:26:57.859158 9395 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:26:57.866825 9395 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:26:57.866952 9395 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:26:57.867246 9395 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.161.2
I20260810 03:26:57.877923 9395 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.161.20:36371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786332336319544-28292-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786332336319544-28292-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786332336319544-28292-0/krb5kdc/oryx_127.27.161.2.keytab
--principal=oryx/127.27.161.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.161.2:39655
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786332336319544-28292-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.161.2
--webserver_port=34099
--tserver_master_addrs=127.27.161.62:42221
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.161.2
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786332336319544-28292-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:26:57.879420 9395 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:26:57.880858 9395 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:26:57.897696 9407 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:26:57.898154 9405 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:26:57.898480 9411 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:26:57.899088 9395 server_base.cc:1034] running on GCE node
I20260810 03:26:57.899781 9395 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:26:57.900803 9395 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:26:57.902050 9395 hybrid_clock.cc:648] HybridClock initialized: now 1786332417901974 us; error 73 us; skew 500 ppm
I20260810 03:26:57.902374 9395 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:26:57 dist-test-slave-vpvm krb5kdc[8293](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332417, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.161.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:26:57.905586 9395 init.cc:377] Logged in from keytab as oryx/127.27.161.2@KRBTEST.COM (short username oryx)
I20260810 03:26:57.907302 9395 webserver.cc:460] Webserver started at http://127.27.161.2:34099/ using document root <none> and password file <none>
I20260810 03:26:57.908004 9395 fs_manager.cc:359] Metadata directory not provided
I20260810 03:26:57.908205 9395 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:26:57.914167 9395 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260810 03:26:57.917249 9419 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:26:57.918861 9395 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 03:26:57.919039 9395 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786332336319544-28292-0/minicluster-data/ts-1/data,/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786332336319544-28292-0/minicluster-data/ts-1/wal
uuid: "2b27714d2680479984776b4b44aff0d9"
format_stamp: "Formatted at 2026-08-10 03:26:55 on dist-test-slave-vpvm"
server_key: "ae6d67ea236403936aeb53943b36d0d8"
server_key_iv: "e43241d3e2e0fb5694481bfced0635d4"
server_key_version: "encryptionkey@0"
I20260810 03:26:57.920006 9395 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786332336319544-28292-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786332336319544-28292-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786332336319544-28292-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:26:57.940685 9395 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:26:57.945154 9395 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:26:57.945431 9395 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:26:57.946802 9395 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:26:57.948367 9395 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:26:57.948484 9395 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:26:57.948591 9395 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:26:57.948670 9395 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:26:57.987929 9395 rpc_server.cc:307] RPC server started. Bound to: 127.27.161.2:39655
I20260810 03:26:57.987977 9592 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.161.2:39655 every 8 connection(s)
I20260810 03:26:57.989428 9395 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786332336319544-28292-0/minicluster-data/ts-1/data/info.pb
I20260810 03:26:57.998898 28292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iBH5/build/asan/bin/kudu as pid 9395
I20260810 03:26:57.999697 28292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iBH5/build/asan/bin/kudu
/tmp/dist-test-task24iBH5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786332336319544-28292-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786332336319544-28292-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786332336319544-28292-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786332336319544-28292-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.161.3:35967
--local_ip_for_outbound_sockets=127.27.161.3
--tserver_master_addrs=127.27.161.62:42221
--webserver_port=41747
--webserver_interface=127.27.161.3
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.3.keytab
--principal=kudu/127.27.161.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.161.20:36371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.27.161.3
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786332336319544-28292-0/krb5kdc/oryx_127.27.161.3.keytab with env {KRB5CCNAME=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786332336319544-28292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786332336319544-28292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786332336319544-28292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 03:26:58 dist-test-slave-vpvm krb5kdc[8293](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.161.2@KRBTEST.COM for oryx/127.27.161.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 03:26:58 dist-test-slave-vpvm krb5kdc[8293](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.161.2@KRBTEST.COM for oryx/127.27.161.62@KRBTEST.COM, Server not found in Kerberos database
W20260810 03:26:58.474470 9605 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:26:58.474996 9605 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:26:58.475199 9605 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:26:58.488865 9605 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:26:58.489053 9605 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:26:58.489295 9605 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.161.3
I20260810 03:26:58.507865 9605 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.161.20:36371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786332336319544-28292-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786332336319544-28292-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786332336319544-28292-0/krb5kdc/oryx_127.27.161.3.keytab
--principal=oryx/127.27.161.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.161.3:35967
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786332336319544-28292-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.161.3
--webserver_port=41747
--tserver_master_addrs=127.27.161.62:42221
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.161.3
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786332336319544-28292-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:26:58.510140 9605 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:26:58.511703 9605 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:26:58.527884 9618 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:26:58.528745 9621 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:26:58.528803 9605 server_base.cc:1034] running on GCE node
W20260810 03:26:58.529583 9616 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:26:58.530405 9605 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:26:58.531466 9605 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:26:58.532687 9605 hybrid_clock.cc:648] HybridClock initialized: now 1786332418532616 us; error 80 us; skew 500 ppm
I20260810 03:26:58.533003 9605 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:26:58 dist-test-slave-vpvm krb5kdc[8293](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332418, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.161.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:26:58.536830 9605 init.cc:377] Logged in from keytab as oryx/127.27.161.3@KRBTEST.COM (short username oryx)
I20260810 03:26:58.538727 9605 webserver.cc:460] Webserver started at http://127.27.161.3:41747/ using document root <none> and password file <none>
I20260810 03:26:58.539486 9605 fs_manager.cc:359] Metadata directory not provided
I20260810 03:26:58.539633 9605 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:26:58.545748 9605 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260810 03:26:58.549017 9631 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:26:58.550525 9605 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 03:26:58.550693 9605 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786332336319544-28292-0/minicluster-data/ts-2/data,/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786332336319544-28292-0/minicluster-data/ts-2/wal
uuid: "8a828ad0fd024a97aec877395b55b090"
format_stamp: "Formatted at 2026-08-10 03:26:55 on dist-test-slave-vpvm"
server_key: "6ae662c75abdc5215012053e41537e52"
server_key_iv: "dac77c5099396acfcf409c80b7dc920d"
server_key_version: "encryptionkey@0"
I20260810 03:26:58.551692 9605 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786332336319544-28292-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786332336319544-28292-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786332336319544-28292-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:26:58.582988 9605 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:26:58.587406 9605 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:26:58.587679 9605 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:26:58.588770 9605 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:26:58.590272 9605 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:26:58.590369 9605 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:26:58.590479 9605 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:26:58.590556 9605 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:26:58.630899 9605 rpc_server.cc:307] RPC server started. Bound to: 127.27.161.3:35967
I20260810 03:26:58.630947 9816 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.161.3:35967 every 8 connection(s)
I20260810 03:26:58.633091 9605 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786332336319544-28292-0/minicluster-data/ts-2/data/info.pb
I20260810 03:26:58.634089 28292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iBH5/build/asan/bin/kudu as pid 9605
Aug 10 03:26:58 dist-test-slave-vpvm krb5kdc[8293](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.161.3@KRBTEST.COM for oryx/127.27.161.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 03:26:58 dist-test-slave-vpvm krb5kdc[8293](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.161.3@KRBTEST.COM for oryx/127.27.161.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 03:26:58.677309 9824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:58.635322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41310 (local address 127.27.161.62:42221)
0810 03:26:58.655685 (+ 20363us) server_negotiation.cc:206] Beginning negotiation
0810 03:26:58.655693 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 03:26:58.655723 (+ 30us) server_negotiation.cc:407] Connection header received
0810 03:26:58.655806 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:26:58.655813 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:26:58.656159 (+ 346us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:26:58.656318 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:26:58.658737 (+ 2419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:58.659894 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:58.662053 (+ 2159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:58.662721 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:58.662844 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:58.666925 (+ 4081us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:26:58.666937 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:26:58.666961 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:26:58.667014 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:26:58.671500 (+ 4486us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:26:58.672441 (+ 941us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:26:58.672452 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:26:58.672462 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:26:58.672701 (+ 239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:26:58.673120 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:26:58.673125 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:26:58.673129 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:26:58.673804 (+ 675us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:26:58.674139 (+ 335us) server_negotiation.cc:1035] Waiting for connection context
0810 03:26:58.675632 (+ 1493us) server_negotiation.cc:299] Negotiation successful
0810 03:26:58.676152 (+ 520us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20179,"thread_start_us":20610,"threads_started":1}
W20260810 03:26:58.748485 9394 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.233s user 0.004s sys 0.012s
W20260810 03:26:58.750440 9393 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:26:57.503462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.161.1:58279 (local address 127.27.161.62:42221)
0810 03:26:57.504305 (+ 843us) server_negotiation.cc:206] Beginning negotiation
0810 03:26:57.504322 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0810 03:26:57.504608 (+ 286us) server_negotiation.cc:407] Connection header received
0810 03:26:57.507363 (+ 2755us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:26:57.507377 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:26:57.507751 (+ 374us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:26:57.508062 (+ 311us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:26:57.510067 (+ 2005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:57.511352 (+ 1285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:57.513470 (+ 2118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:57.514222 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:57.514366 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:58.750053 (+1235687us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.161.1:58279: BlockingRecv error: recv got EOF from 127.27.161.1:58279 (error 108)
Metrics: {"server-negotiator.queue_time_us":552,"thread_start_us":299,"threads_started":1}
I20260810 03:26:58.751075 9394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:57.503508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:42221 (local address 127.27.161.1:58279)
0810 03:26:57.504384 (+ 876us) negotiation.cc:107] Waiting for socket to connect
0810 03:26:57.504466 (+ 82us) client_negotiation.cc:174] Beginning negotiation
0810 03:26:57.506727 (+ 2261us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:26:57.508325 (+ 1598us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:26:57.508337 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:26:57.509046 (+ 709us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:26:57.509817 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:26:57.509842 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:26:57.511542 (+ 1700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:26:57.511547 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:26:57.513304 (+ 1757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:26:57.513321 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:26:57.514466 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:26:57.514473 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:26:57.514773 (+ 300us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:57.515522 (+ 749us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:26:57.515552 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:26:58.749619 (+1234067us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.161.62:42221: Server oryx/127.27.161.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":549,"thread_start_us":314,"threads_started":1}
W20260810 03:26:58.752574 9389 heartbeater.cc:644] Failed to heartbeat to 127.27.161.62:42221 (0 consecutive failures): Not authorized: Failed to ping master at 127.27.161.62:42221: Client connection negotiation failed: client connection to 127.27.161.62:42221: Server oryx/127.27.161.62@KRBTEST.COM not found in Kerberos database
Aug 10 03:26:58 dist-test-slave-vpvm krb5kdc[8293](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.161.1@KRBTEST.COM for oryx/127.27.161.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 03:26:58 dist-test-slave-vpvm krb5kdc[8293](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.161.1@KRBTEST.COM for oryx/127.27.161.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 03:26:58.768023 9394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:58.753443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:42221 (local address 127.27.161.1:43017)
0810 03:26:58.753689 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0810 03:26:58.753717 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0810 03:26:58.753876 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:26:58.754667 (+ 791us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:26:58.754673 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:26:58.755272 (+ 599us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:26:58.755547 (+ 275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:26:58.755562 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:26:58.757412 (+ 1850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:26:58.757419 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:26:58.759221 (+ 1802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:26:58.759241 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:26:58.760693 (+ 1452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:26:58.760701 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:26:58.760924 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:58.761848 (+ 924us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:26:58.761875 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:26:58.767559 (+ 5684us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.161.62:42221: Server oryx/127.27.161.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":72}
W20260810 03:26:58.768208 9393 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:26:58.754020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.161.1:43017 (local address 127.27.161.62:42221)
0810 03:26:58.754258 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0810 03:26:58.754265 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:26:58.754287 (+ 22us) server_negotiation.cc:407] Connection header received
0810 03:26:58.754361 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:26:58.754368 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:26:58.754459 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:26:58.754575 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:26:58.755739 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:58.757142 (+ 1403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:58.759443 (+ 2301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:58.760214 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:58.760328 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:58.767921 (+ 7593us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.161.1:43017: BlockingRecv error: recv got EOF from 127.27.161.1:43017 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Aug 10 03:26:58 dist-test-slave-vpvm krb5kdc[8293](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.161.1@KRBTEST.COM for oryx/127.27.161.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 03:26:58 dist-test-slave-vpvm krb5kdc[8293](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.161.1@KRBTEST.COM for oryx/127.27.161.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 03:26:58.789515 9394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:58.769405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:42221 (local address 127.27.161.1:33809)
0810 03:26:58.774280 (+ 4875us) negotiation.cc:107] Waiting for socket to connect
0810 03:26:58.774308 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0810 03:26:58.774541 (+ 233us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:26:58.775168 (+ 627us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:26:58.775174 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:26:58.775797 (+ 623us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:26:58.776087 (+ 290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:26:58.776100 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:26:58.777897 (+ 1797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:26:58.777903 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:26:58.779815 (+ 1912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:26:58.779834 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:26:58.781256 (+ 1422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:26:58.781263 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:26:58.781550 (+ 287us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:58.782982 (+ 1432us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:26:58.783039 (+ 57us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:26:58.788993 (+ 5954us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.161.62:42221: Server oryx/127.27.161.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":4730,"thread_start_us":12857,"threads_started":1}
W20260810 03:26:58.789661 9393 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:26:58.769919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.161.1:33809 (local address 127.27.161.62:42221)
0810 03:26:58.770164 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0810 03:26:58.770172 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 03:26:58.774404 (+ 4232us) server_negotiation.cc:407] Connection header received
0810 03:26:58.774715 (+ 311us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:26:58.774721 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:26:58.774833 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:26:58.774908 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:26:58.776321 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:58.777636 (+ 1315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:58.780027 (+ 2391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:58.780822 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:58.780938 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:58.789313 (+ 8375us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.161.1:33809: BlockingRecv error: recv got EOF from 127.27.161.1:33809 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
W20260810 03:26:58.790045 9389 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260810 03:26:59.186647 9598 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.148s user 0.001s sys 0.019s
W20260810 03:26:59.188156 9597 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:26:57.992205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.161.2:33087 (local address 127.27.161.62:42221)
0810 03:26:58.008199 (+ 15994us) server_negotiation.cc:206] Beginning negotiation
0810 03:26:58.008210 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 03:26:58.024788 (+ 16578us) server_negotiation.cc:407] Connection header received
0810 03:26:58.028455 (+ 3667us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:26:58.028462 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:26:58.028589 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:26:58.028710 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:26:58.031752 (+ 3042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:58.032983 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:58.036029 (+ 3046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:58.036703 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:58.036863 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:59.187830 (+1150967us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.161.2:33087: BlockingRecv error: recv got EOF from 127.27.161.2:33087 (error 108)
Metrics: {"server-negotiator.queue_time_us":15740,"thread_start_us":16013,"threads_started":1}
I20260810 03:26:59.188787 9598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:57.992780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:42221 (local address 127.27.161.2:33087)
0810 03:26:58.024374 (+ 31594us) negotiation.cc:107] Waiting for socket to connect
0810 03:26:58.024568 (+ 194us) client_negotiation.cc:174] Beginning negotiation
0810 03:26:58.027926 (+ 3358us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:26:58.029146 (+ 1220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:26:58.029164 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:26:58.030238 (+ 1074us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:26:58.031435 (+ 1197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:26:58.031473 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:26:58.033199 (+ 1726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:26:58.033206 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:26:58.035789 (+ 2583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:26:58.035816 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:26:58.037066 (+ 1250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:26:58.037076 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:26:58.037483 (+ 407us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:58.038531 (+ 1048us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:26:58.038587 (+ 56us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:26:59.187649 (+1149062us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.161.62:42221: Server oryx/127.27.161.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":31071,"thread_start_us":30725,"threads_started":1}
W20260810 03:26:59.189805 9593 heartbeater.cc:644] Failed to heartbeat to 127.27.161.62:42221 (0 consecutive failures): Not authorized: Failed to ping master at 127.27.161.62:42221: Client connection negotiation failed: client connection to 127.27.161.62:42221: Server oryx/127.27.161.62@KRBTEST.COM not found in Kerberos database
Aug 10 03:26:59 dist-test-slave-vpvm krb5kdc[8293](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.161.2@KRBTEST.COM for oryx/127.27.161.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 03:26:59 dist-test-slave-vpvm krb5kdc[8293](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.161.2@KRBTEST.COM for oryx/127.27.161.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 03:26:59.199712 9598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:59.190629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:42221 (local address 127.27.161.2:38827)
0810 03:26:59.190874 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0810 03:26:59.190894 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 03:26:59.191033 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:26:59.191376 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:26:59.191381 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:26:59.191783 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:26:59.191962 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:26:59.191972 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:26:59.193037 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:26:59.193041 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:26:59.194074 (+ 1033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:26:59.194088 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:26:59.194808 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:26:59.194815 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:26:59.194978 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:59.195590 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:26:59.195609 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:26:59.199406 (+ 3797us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.161.62:42221: Server oryx/127.27.161.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":79}
W20260810 03:26:59.199857 9597 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:26:59.190706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.161.2:38827 (local address 127.27.161.62:42221)
0810 03:26:59.190862 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0810 03:26:59.190867 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:26:59.190954 (+ 87us) server_negotiation.cc:407] Connection header received
0810 03:26:59.191162 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:26:59.191168 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:26:59.191241 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:26:59.191325 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:26:59.192116 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:59.192884 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:59.194216 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:59.194623 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:59.194741 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:59.199620 (+ 4879us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.161.2:38827: BlockingRecv error: recv got EOF from 127.27.161.2:38827 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Aug 10 03:26:59 dist-test-slave-vpvm krb5kdc[8293](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.161.2@KRBTEST.COM for oryx/127.27.161.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 03:26:59 dist-test-slave-vpvm krb5kdc[8293](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.161.2@KRBTEST.COM for oryx/127.27.161.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 03:26:59.209383 9598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:59.200842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:42221 (local address 127.27.161.2:39777)
0810 03:26:59.201008 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0810 03:26:59.201026 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 03:26:59.201119 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:26:59.201566 (+ 447us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:26:59.201571 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:26:59.201913 (+ 342us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:26:59.202139 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:26:59.202150 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:26:59.203180 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:26:59.203186 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:26:59.204231 (+ 1045us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:26:59.204246 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:26:59.204935 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:26:59.204941 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:26:59.205145 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:59.205709 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:26:59.205728 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:26:59.209088 (+ 3360us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.161.62:42221: Server oryx/127.27.161.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":49}
W20260810 03:26:59.209630 9597 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:26:59.201088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.161.2:39777 (local address 127.27.161.62:42221)
0810 03:26:59.201280 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0810 03:26:59.201284 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:26:59.201298 (+ 14us) server_negotiation.cc:407] Connection header received
0810 03:26:59.201354 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:26:59.201358 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:26:59.201422 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:26:59.201505 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:26:59.202281 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:59.203027 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:59.204373 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:59.204781 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:59.204916 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:59.209360 (+ 4444us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.161.2:39777: BlockingRecv error: recv got EOF from 127.27.161.2:39777 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
W20260810 03:26:59.209857 9593 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260810 03:26:59.687758 9830 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.021s user 0.004s sys 0.013s
W20260810 03:26:59.689065 9829 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:26:58.650347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.161.3:41043 (local address 127.27.161.62:42221)
0810 03:26:58.657935 (+ 7588us) server_negotiation.cc:206] Beginning negotiation
0810 03:26:58.657940 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:26:58.657978 (+ 38us) server_negotiation.cc:407] Connection header received
0810 03:26:58.658029 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:26:58.658034 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:26:58.658106 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:26:58.658225 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:26:58.660576 (+ 2351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:58.661749 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:58.664235 (+ 2486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:58.664864 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:58.664959 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:59.688815 (+1023856us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.161.3:41043: BlockingRecv error: recv got EOF from 127.27.161.3:41043 (error 108)
Metrics: {"server-negotiator.queue_time_us":7442,"thread_start_us":7609,"threads_started":1}
I20260810 03:26:59.689644 9830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:58.651141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:42221 (local address 127.27.161.3:41043)
0810 03:26:58.654581 (+ 3440us) negotiation.cc:107] Waiting for socket to connect
0810 03:26:58.654633 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0810 03:26:58.657221 (+ 2588us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:26:58.658736 (+ 1515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:26:58.658749 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:26:58.659519 (+ 770us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:26:58.660363 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:26:58.660393 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:26:58.661980 (+ 1587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:26:58.661985 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:26:58.664043 (+ 2058us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:26:58.664060 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:26:58.665355 (+ 1295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:26:58.665364 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:26:58.665677 (+ 313us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:58.666431 (+ 754us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:26:58.666464 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:26:59.688583 (+1022119us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.161.62:42221: Server oryx/127.27.161.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":3114,"thread_start_us":2870,"threads_started":1}
W20260810 03:26:59.690644 9821 heartbeater.cc:644] Failed to heartbeat to 127.27.161.62:42221 (0 consecutive failures): Not authorized: Failed to ping master at 127.27.161.62:42221: Client connection negotiation failed: client connection to 127.27.161.62:42221: Server oryx/127.27.161.62@KRBTEST.COM not found in Kerberos database
Aug 10 03:26:59 dist-test-slave-vpvm krb5kdc[8293](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.161.3@KRBTEST.COM for oryx/127.27.161.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 03:26:59 dist-test-slave-vpvm krb5kdc[8293](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.161.3@KRBTEST.COM for oryx/127.27.161.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 03:26:59.700268 9830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:59.691316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:42221 (local address 127.27.161.3:33761)
0810 03:26:59.691519 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0810 03:26:59.691540 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0810 03:26:59.691653 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:26:59.691994 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:26:59.691998 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:26:59.692375 (+ 377us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:26:59.692563 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:26:59.692572 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:26:59.693666 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:26:59.693673 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:26:59.694799 (+ 1126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:26:59.694815 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:26:59.695618 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:26:59.695629 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:26:59.695766 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:59.696357 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:26:59.696378 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:26:59.699950 (+ 3572us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.161.62:42221: Server oryx/127.27.161.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":57}
W20260810 03:26:59.700394 9829 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:26:59.691375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.161.3:33761 (local address 127.27.161.62:42221)
0810 03:26:59.691532 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0810 03:26:59.691536 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:26:59.691596 (+ 60us) server_negotiation.cc:407] Connection header received
0810 03:26:59.691762 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:26:59.691765 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:26:59.691829 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:26:59.691903 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:26:59.692720 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:59.693458 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:59.694968 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:59.695402 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:59.695509 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:59.700168 (+ 4659us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.161.3:33761: BlockingRecv error: recv got EOF from 127.27.161.3:33761 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Aug 10 03:26:59 dist-test-slave-vpvm krb5kdc[8293](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.161.3@KRBTEST.COM for oryx/127.27.161.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 03:26:59 dist-test-slave-vpvm krb5kdc[8293](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.161.3@KRBTEST.COM for oryx/127.27.161.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 03:26:59.708981 9830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:59.701014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:42221 (local address 127.27.161.3:59579)
0810 03:26:59.701187 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0810 03:26:59.701208 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0810 03:26:59.701322 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:26:59.701624 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:26:59.701627 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:26:59.701937 (+ 310us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:26:59.702148 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:26:59.702158 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:26:59.703098 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:26:59.703102 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:26:59.704086 (+ 984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:26:59.704100 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:26:59.704771 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:26:59.704775 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:26:59.704912 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:59.705477 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:26:59.705495 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:26:59.708703 (+ 3208us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.161.62:42221: Server oryx/127.27.161.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":48}
W20260810 03:26:59.709200 9829 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:26:59.701125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.161.3:59579 (local address 127.27.161.62:42221)
0810 03:26:59.701369 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0810 03:26:59.701373 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:26:59.701387 (+ 14us) server_negotiation.cc:407] Connection header received
0810 03:26:59.701429 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:26:59.701432 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:26:59.701503 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:26:59.701578 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:26:59.702273 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:59.702980 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:59.704202 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:59.704649 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:59.704739 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:59.708943 (+ 4204us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.161.3:59579: BlockingRecv error: recv got EOF from 127.27.161.3:59579 (error 108)
Metrics: {"server-negotiator.queue_time_us":142}
W20260810 03:26:59.709345 9821 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Aug 10 03:26:59 dist-test-slave-vpvm krb5kdc[8293](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.161.1@KRBTEST.COM for oryx/127.27.161.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 03:26:59 dist-test-slave-vpvm krb5kdc[8293](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.161.1@KRBTEST.COM for oryx/127.27.161.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 03:26:59.799785 9834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:26:59.790614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:42221 (local address 127.27.161.1:37077)
0810 03:26:59.791182 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0810 03:26:59.791206 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0810 03:26:59.791327 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:26:59.791634 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:26:59.791637 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:26:59.792066 (+ 429us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:26:59.792233 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:26:59.792242 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:26:59.793283 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:26:59.793288 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:26:59.794339 (+ 1051us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:26:59.794354 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:26:59.795010 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:26:59.795015 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:26:59.795148 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:59.795755 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:26:59.795772 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:26:59.799539 (+ 3767us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.161.62:42221: Server oryx/127.27.161.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":339,"threads_started":1}
W20260810 03:26:59.799947 9829 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:26:59.790677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.161.1:37077 (local address 127.27.161.62:42221)
0810 03:26:59.790836 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0810 03:26:59.790840 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:26:59.791274 (+ 434us) server_negotiation.cc:407] Connection header received
0810 03:26:59.791419 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:26:59.791423 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:26:59.791487 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:26:59.791566 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:26:59.792379 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:59.793099 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:59.794452 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:26:59.794866 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:26:59.794955 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:26:59.799753 (+ 4798us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.161.1:37077: BlockingRecv error: recv got EOF from 127.27.161.1:37077 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Aug 10 03:27:00 dist-test-slave-vpvm krb5kdc[8293](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.161.2@KRBTEST.COM for oryx/127.27.161.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 03:27:00 dist-test-slave-vpvm krb5kdc[8293](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.161.2@KRBTEST.COM for oryx/127.27.161.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 03:27:00.220293 9835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:27:00.210721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:42221 (local address 127.27.161.2:34375)
0810 03:27:00.211308 (+ 587us) negotiation.cc:107] Waiting for socket to connect
0810 03:27:00.211326 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 03:27:00.211449 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:27:00.211841 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:27:00.211848 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:27:00.212451 (+ 603us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:27:00.212635 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:27:00.212647 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:27:00.213734 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:27:00.213739 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:27:00.214742 (+ 1003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:27:00.214756 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:27:00.215755 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:27:00.215761 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:27:00.215895 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:27:00.216517 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:27:00.216537 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:27:00.220032 (+ 3495us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.161.62:42221: Server oryx/127.27.161.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":320,"threads_started":1}
W20260810 03:27:00.220525 9829 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:27:00.210890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.161.2:34375 (local address 127.27.161.62:42221)
0810 03:27:00.211147 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0810 03:27:00.211152 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:27:00.211413 (+ 261us) server_negotiation.cc:407] Connection header received
0810 03:27:00.211561 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:27:00.211566 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:27:00.211666 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:27:00.211753 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:27:00.212808 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:27:00.213573 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:27:00.214904 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:27:00.215544 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:27:00.215660 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:27:00.220270 (+ 4610us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.161.2:34375: BlockingRecv error: recv got EOF from 127.27.161.2:34375 (error 108)
Metrics: {"server-negotiator.queue_time_us":100}
Aug 10 03:27:00 dist-test-slave-vpvm krb5kdc[8293](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.161.3@KRBTEST.COM for oryx/127.27.161.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 03:27:00 dist-test-slave-vpvm krb5kdc[8293](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.161.3@KRBTEST.COM for oryx/127.27.161.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 03:27:00.719359 9837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:27:00.709919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:42221 (local address 127.27.161.3:43739)
0810 03:27:00.710542 (+ 623us) negotiation.cc:107] Waiting for socket to connect
0810 03:27:00.710558 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 03:27:00.710700 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:27:00.711035 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:27:00.711040 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:27:00.711503 (+ 463us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:27:00.711661 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:27:00.711670 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:27:00.712718 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:27:00.712722 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:27:00.713695 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:27:00.713707 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:27:00.714544 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:27:00.714551 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:27:00.714712 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:27:00.715297 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:27:00.715316 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:27:00.719054 (+ 3738us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.161.62:42221: Server oryx/127.27.161.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":524,"thread_start_us":337,"threads_started":1}
W20260810 03:27:00.719483 9829 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:27:00.710160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.161.3:43739 (local address 127.27.161.62:42221)
0810 03:27:00.710387 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0810 03:27:00.710393 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:27:00.710646 (+ 253us) server_negotiation.cc:407] Connection header received
0810 03:27:00.710813 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:27:00.710818 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:27:00.710902 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:27:00.710983 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:27:00.711812 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:27:00.712564 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:27:00.713843 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:27:00.714359 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:27:00.714474 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:27:00.719273 (+ 4799us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.161.3:43739: BlockingRecv error: recv got EOF from 127.27.161.3:43739 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Aug 10 03:27:00 dist-test-slave-vpvm krb5kdc[8293](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.161.1@KRBTEST.COM for oryx/127.27.161.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 03:27:00 dist-test-slave-vpvm krb5kdc[8293](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.161.1@KRBTEST.COM for oryx/127.27.161.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 03:27:00.811487 9838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:27:00.800848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:42221 (local address 127.27.161.1:40325)
0810 03:27:00.801342 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0810 03:27:00.801363 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0810 03:27:00.801496 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:27:00.801893 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:27:00.801897 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:27:00.802401 (+ 504us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:27:00.802603 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:27:00.802613 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:27:00.803821 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:27:00.803827 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:27:00.804857 (+ 1030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:27:00.804874 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:27:00.805664 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:27:00.805669 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:27:00.805814 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:27:00.806540 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:27:00.806564 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:27:00.811100 (+ 4536us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.161.62:42221: Server oryx/127.27.161.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":261,"threads_started":1}
W20260810 03:27:00.811614 9829 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:27:00.800990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.161.1:40325 (local address 127.27.161.62:42221)
0810 03:27:00.801260 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0810 03:27:00.801267 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:27:00.801482 (+ 215us) server_negotiation.cc:407] Connection header received
0810 03:27:00.801629 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:27:00.801633 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:27:00.801704 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:27:00.801789 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:27:00.802787 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:27:00.803653 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:27:00.805042 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:27:00.805473 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:27:00.805587 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:27:00.811367 (+ 5780us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.161.1:40325: BlockingRecv error: recv got EOF from 127.27.161.1:40325 (error 108)
Metrics: {"server-negotiator.queue_time_us":103}
Aug 10 03:27:01 dist-test-slave-vpvm krb5kdc[8293](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.161.2@KRBTEST.COM for oryx/127.27.161.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 03:27:01 dist-test-slave-vpvm krb5kdc[8293](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.161.2@KRBTEST.COM for oryx/127.27.161.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 03:27:01.231042 9839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:27:01.221460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:42221 (local address 127.27.161.2:41745)
0810 03:27:01.222064 (+ 604us) negotiation.cc:107] Waiting for socket to connect
0810 03:27:01.222083 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 03:27:01.222218 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:27:01.222754 (+ 536us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:27:01.222759 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:27:01.223254 (+ 495us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:27:01.223452 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:27:01.223485 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:27:01.224671 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:27:01.224677 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:27:01.225910 (+ 1233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:27:01.225928 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:27:01.226654 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:27:01.226659 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:27:01.226786 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:27:01.227346 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:27:01.227363 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:27:01.230770 (+ 3407us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.161.62:42221: Server oryx/127.27.161.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":311,"threads_started":1}
W20260810 03:27:01.231170 9829 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:27:01.221709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.161.2:41745 (local address 127.27.161.62:42221)
0810 03:27:01.222419 (+ 710us) server_negotiation.cc:206] Beginning negotiation
0810 03:27:01.222424 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:27:01.222444 (+ 20us) server_negotiation.cc:407] Connection header received
0810 03:27:01.222505 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:27:01.222509 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:27:01.222576 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:27:01.222669 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:27:01.223648 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:27:01.224506 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:27:01.226079 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:27:01.226510 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:27:01.226610 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:27:01.230979 (+ 4369us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.161.2:41745: BlockingRecv error: recv got EOF from 127.27.161.2:41745 (error 108)
Metrics: {"server-negotiator.queue_time_us":565}
Aug 10 03:27:01 dist-test-slave-vpvm krb5kdc[8293](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.161.3@KRBTEST.COM for oryx/127.27.161.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 03:27:01 dist-test-slave-vpvm krb5kdc[8293](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.161.3@KRBTEST.COM for oryx/127.27.161.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 03:27:01.729790 9840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:27:01.720469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:42221 (local address 127.27.161.3:57823)
0810 03:27:01.721018 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0810 03:27:01.721035 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 03:27:01.721175 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:27:01.721549 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:27:01.721554 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:27:01.722058 (+ 504us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:27:01.722268 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:27:01.722280 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:27:01.723409 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:27:01.723413 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:27:01.724425 (+ 1012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:27:01.724441 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:27:01.725138 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:27:01.725144 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:27:01.725313 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:27:01.725909 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:27:01.725927 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:27:01.729492 (+ 3565us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.161.62:42221: Server oryx/127.27.161.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":301,"threads_started":1}
W20260810 03:27:01.730017 9829 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:27:01.720697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.161.3:57823 (local address 127.27.161.62:42221)
0810 03:27:01.720923 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0810 03:27:01.720930 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:27:01.721112 (+ 182us) server_negotiation.cc:407] Connection header received
0810 03:27:01.721306 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:27:01.721310 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:27:01.721385 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:27:01.721468 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:27:01.722418 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:27:01.723256 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:27:01.724577 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:27:01.724970 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:27:01.725103 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:27:01.729735 (+ 4632us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.161.3:57823: BlockingRecv error: recv got EOF from 127.27.161.3:57823 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Aug 10 03:27:01 dist-test-slave-vpvm krb5kdc[8293](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.161.1@KRBTEST.COM for oryx/127.27.161.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 03:27:01 dist-test-slave-vpvm krb5kdc[8293](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.161.1@KRBTEST.COM for oryx/127.27.161.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 03:27:01.822530 9841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:27:01.812859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:42221 (local address 127.27.161.1:45943)
0810 03:27:01.813382 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0810 03:27:01.813404 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0810 03:27:01.813532 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:27:01.813907 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:27:01.813910 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:27:01.814338 (+ 428us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:27:01.814535 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:27:01.814546 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:27:01.815767 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:27:01.815772 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:27:01.816724 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:27:01.816736 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:27:01.817623 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:27:01.817628 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:27:01.817766 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:27:01.818476 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:27:01.818495 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:27:01.822269 (+ 3774us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.161.62:42221: Server oryx/127.27.161.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":277,"threads_started":1}
W20260810 03:27:01.822866 9829 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:27:01.812962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.161.1:45943 (local address 127.27.161.62:42221)
0810 03:27:01.813170 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0810 03:27:01.813175 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:27:01.813495 (+ 320us) server_negotiation.cc:407] Connection header received
0810 03:27:01.813649 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:27:01.813653 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:27:01.813722 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:27:01.813814 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:27:01.814735 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:27:01.815515 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:27:01.816886 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:27:01.817454 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:27:01.817593 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:27:01.822517 (+ 4924us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.161.1:45943: BlockingRecv error: recv got EOF from 127.27.161.1:45943 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
Aug 10 03:27:02 dist-test-slave-vpvm krb5kdc[8293](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.161.2@KRBTEST.COM for oryx/127.27.161.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 03:27:02 dist-test-slave-vpvm krb5kdc[8293](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.161.2@KRBTEST.COM for oryx/127.27.161.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 03:27:02.241454 9842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:27:02.232187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:42221 (local address 127.27.161.2:56225)
0810 03:27:02.232791 (+ 604us) negotiation.cc:107] Waiting for socket to connect
0810 03:27:02.232812 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0810 03:27:02.232957 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:27:02.233328 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:27:02.233333 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:27:02.233812 (+ 479us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:27:02.234050 (+ 238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:27:02.234061 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:27:02.235134 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:27:02.235149 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:27:02.236206 (+ 1057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:27:02.236220 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:27:02.236921 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:27:02.236927 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:27:02.237116 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:27:02.237722 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:27:02.237739 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:27:02.241115 (+ 3376us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.161.62:42221: Server oryx/127.27.161.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":298,"threads_started":1}
W20260810 03:27:02.241636 9829 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:27:02.232358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.161.2:56225 (local address 127.27.161.62:42221)
0810 03:27:02.232574 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0810 03:27:02.232581 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:27:02.232947 (+ 366us) server_negotiation.cc:407] Connection header received
0810 03:27:02.233074 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:27:02.233080 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:27:02.233163 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:27:02.233268 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:27:02.234220 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:27:02.234976 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:27:02.236344 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:27:02.236762 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:27:02.236897 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:27:02.241388 (+ 4491us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.161.2:56225: BlockingRecv error: recv got EOF from 127.27.161.2:56225 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Aug 10 03:27:02 dist-test-slave-vpvm krb5kdc[8293](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.161.3@KRBTEST.COM for oryx/127.27.161.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 03:27:02 dist-test-slave-vpvm krb5kdc[8293](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.161.3@KRBTEST.COM for oryx/127.27.161.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 03:27:02.743183 9845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:27:02.731002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:42221 (local address 127.27.161.3:59505)
0810 03:27:02.731676 (+ 674us) negotiation.cc:107] Waiting for socket to connect
0810 03:27:02.731693 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 03:27:02.731822 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:27:02.732124 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:27:02.732127 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:27:02.732503 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:27:02.732686 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:27:02.732696 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:27:02.733679 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:27:02.733682 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:27:02.734634 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:27:02.734656 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:27:02.735293 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:27:02.735297 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:27:02.735418 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:27:02.739056 (+ 3638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:27:02.739078 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:27:02.742911 (+ 3833us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.161.62:42221: Server oryx/127.27.161.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":539,"thread_start_us":401,"threads_started":1}
W20260810 03:27:02.743402 9829 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:27:02.731169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.161.3:59505 (local address 127.27.161.62:42221)
0810 03:27:02.731379 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0810 03:27:02.731383 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:27:02.731809 (+ 426us) server_negotiation.cc:407] Connection header received
0810 03:27:02.731916 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:27:02.731920 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:27:02.732004 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:27:02.732090 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:27:02.732834 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:27:02.733548 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:27:02.734755 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:27:02.735186 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:27:02.735275 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:27:02.743148 (+ 7873us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.161.3:59505: BlockingRecv error: recv got EOF from 127.27.161.3:59505 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
Aug 10 03:27:02 dist-test-slave-vpvm krb5kdc[8293](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.161.1@KRBTEST.COM for oryx/127.27.161.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 03:27:02 dist-test-slave-vpvm krb5kdc[8293](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.161.1@KRBTEST.COM for oryx/127.27.161.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 03:27:02.832964 9847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:27:02.823634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:42221 (local address 127.27.161.1:34807)
0810 03:27:02.824168 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0810 03:27:02.824190 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0810 03:27:02.824407 (+ 217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:27:02.824747 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:27:02.824750 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:27:02.825124 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:27:02.825306 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:27:02.825315 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:27:02.826670 (+ 1355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:27:02.826676 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:27:02.827726 (+ 1050us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:27:02.827740 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:27:02.828378 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:27:02.828382 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:27:02.828504 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:27:02.829076 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:27:02.829092 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:27:02.832700 (+ 3608us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.161.62:42221: Server oryx/127.27.161.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":287,"threads_started":1}
W20260810 03:27:02.833220 9829 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:27:02.823756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.161.1:34807 (local address 127.27.161.62:42221)
0810 03:27:02.823934 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0810 03:27:02.823939 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:27:02.824293 (+ 354us) server_negotiation.cc:407] Connection header received
0810 03:27:02.824558 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:27:02.824563 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:27:02.824631 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:27:02.824714 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:27:02.825437 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:27:02.826533 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:27:02.827854 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:27:02.828271 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:27:02.828365 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:27:02.832954 (+ 4589us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.161.1:34807: BlockingRecv error: recv got EOF from 127.27.161.1:34807 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Aug 10 03:27:03 dist-test-slave-vpvm krb5kdc[8293](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.161.2@KRBTEST.COM for oryx/127.27.161.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 03:27:03 dist-test-slave-vpvm krb5kdc[8293](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.161.2@KRBTEST.COM for oryx/127.27.161.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 03:27:03.252882 9849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:27:03.242672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:42221 (local address 127.27.161.2:48015)
0810 03:27:03.243200 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0810 03:27:03.243230 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0810 03:27:03.243352 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:27:03.243690 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:27:03.243693 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:27:03.244111 (+ 418us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:27:03.244288 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:27:03.244299 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:27:03.245333 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:27:03.245337 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:27:03.246506 (+ 1169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:27:03.246522 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:27:03.247268 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:27:03.247273 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:27:03.247422 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:27:03.248246 (+ 824us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:27:03.248273 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:27:03.252562 (+ 4289us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.161.62:42221: Server oryx/127.27.161.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":280,"threads_started":1}
W20260810 03:27:03.253057 9829 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:27:03.242765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.161.2:48015 (local address 127.27.161.62:42221)
0810 03:27:03.243000 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0810 03:27:03.243007 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:27:03.243312 (+ 305us) server_negotiation.cc:407] Connection header received
0810 03:27:03.243454 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:27:03.243460 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:27:03.243545 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:27:03.243652 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:27:03.244414 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:27:03.245186 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:27:03.246639 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:27:03.247111 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:27:03.247208 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:27:03.252827 (+ 5619us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.161.2:48015: BlockingRecv error: recv got EOF from 127.27.161.2:48015 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Aug 10 03:27:03 dist-test-slave-vpvm krb5kdc[8293](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.161.3@KRBTEST.COM for oryx/127.27.161.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 03:27:03 dist-test-slave-vpvm krb5kdc[8293](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.161.3@KRBTEST.COM for oryx/127.27.161.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 03:27:03.757175 9850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:27:03.744286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:42221 (local address 127.27.161.3:51259)
0810 03:27:03.744793 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0810 03:27:03.744810 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 03:27:03.744929 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:27:03.745227 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:27:03.745230 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:27:03.745669 (+ 439us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:27:03.745851 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:27:03.745861 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:27:03.746940 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:27:03.746947 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:27:03.747974 (+ 1027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:27:03.747989 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:27:03.748803 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:27:03.748808 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:27:03.748954 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:27:03.752944 (+ 3990us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:27:03.752965 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:27:03.756906 (+ 3941us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.161.62:42221: Server oryx/127.27.161.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":284,"threads_started":1}
W20260810 03:27:03.757342 9829 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:27:03.744351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.161.3:51259 (local address 127.27.161.62:42221)
0810 03:27:03.744577 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0810 03:27:03.744584 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:27:03.744885 (+ 301us) server_negotiation.cc:407] Connection header received
0810 03:27:03.745037 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:27:03.745041 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:27:03.745122 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:27:03.745200 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:27:03.746020 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:27:03.746778 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:27:03.748182 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:27:03.748641 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:27:03.748742 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:27:03.757115 (+ 8373us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.161.3:51259: BlockingRecv error: recv got EOF from 127.27.161.3:51259 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Aug 10 03:27:03 dist-test-slave-vpvm krb5kdc[8293](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.161.1@KRBTEST.COM for oryx/127.27.161.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 03:27:03 dist-test-slave-vpvm krb5kdc[8293](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.161.1@KRBTEST.COM for oryx/127.27.161.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 03:27:03.844203 9853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:27:03.834181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:42221 (local address 127.27.161.1:54343)
0810 03:27:03.834922 (+ 741us) negotiation.cc:107] Waiting for socket to connect
0810 03:27:03.834942 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 03:27:03.835081 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:27:03.835376 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:27:03.835379 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:27:03.835811 (+ 432us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:27:03.835975 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:27:03.835985 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:27:03.837078 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:27:03.837083 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:27:03.838106 (+ 1023us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:27:03.838120 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:27:03.838921 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:27:03.838950 (+ 29us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:27:03.839099 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:27:03.839974 (+ 875us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:27:03.839994 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:27:03.843751 (+ 3757us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.161.62:42221: Server oryx/127.27.161.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":626,"thread_start_us":488,"threads_started":1}
W20260810 03:27:03.844369 9829 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:27:03.834271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.161.1:54343 (local address 127.27.161.62:42221)
0810 03:27:03.834482 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0810 03:27:03.834486 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:27:03.835056 (+ 570us) server_negotiation.cc:407] Connection header received
0810 03:27:03.835183 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:27:03.835187 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:27:03.835256 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:27:03.835343 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:27:03.836152 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:27:03.836910 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:27:03.838265 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:27:03.838694 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:27:03.838893 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:27:03.844121 (+ 5228us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.161.1:54343: BlockingRecv error: recv got EOF from 127.27.161.1:54343 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Aug 10 03:27:04 dist-test-slave-vpvm krb5kdc[8293](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.161.2@KRBTEST.COM for oryx/127.27.161.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 03:27:04 dist-test-slave-vpvm krb5kdc[8293](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.161.2@KRBTEST.COM for oryx/127.27.161.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 03:27:04.265774 9854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:27:04.253925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:42221 (local address 127.27.161.2:60009)
0810 03:27:04.254501 (+ 576us) negotiation.cc:107] Waiting for socket to connect
0810 03:27:04.254521 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 03:27:04.254683 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:27:04.255086 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:27:04.255091 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:27:04.255618 (+ 527us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:27:04.255831 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:27:04.255844 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:27:04.257850 (+ 2006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:27:04.257856 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:27:04.259323 (+ 1467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:27:04.259342 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:27:04.260386 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:27:04.260392 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:27:04.260577 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:27:04.261278 (+ 701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:27:04.261305 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:27:04.265433 (+ 4128us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.161.62:42221: Server oryx/127.27.161.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":416,"threads_started":1}
W20260810 03:27:04.265928 9829 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:27:04.254162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.161.2:60009 (local address 127.27.161.62:42221)
0810 03:27:04.254373 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0810 03:27:04.254380 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:27:04.254619 (+ 239us) server_negotiation.cc:407] Connection header received
0810 03:27:04.254806 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:27:04.254812 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:27:04.254915 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:27:04.255019 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:27:04.255991 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:27:04.257200 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:27:04.259608 (+ 2408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:27:04.260216 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:27:04.260338 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:27:04.265680 (+ 5342us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.161.2:60009: BlockingRecv error: recv got EOF from 127.27.161.2:60009 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Aug 10 03:27:04 dist-test-slave-vpvm krb5kdc[8293](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.161.3@KRBTEST.COM for oryx/127.27.161.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 03:27:04 dist-test-slave-vpvm krb5kdc[8293](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.161.3@KRBTEST.COM for oryx/127.27.161.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 03:27:04.767385 9858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:27:04.758256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:42221 (local address 127.27.161.3:60295)
0810 03:27:04.758760 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0810 03:27:04.758777 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 03:27:04.758903 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:27:04.759201 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:27:04.759205 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:27:04.759658 (+ 453us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:27:04.759815 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:27:04.759825 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:27:04.760797 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:27:04.760801 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:27:04.761928 (+ 1127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:27:04.761944 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:27:04.762786 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:27:04.762790 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:27:04.762965 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:27:04.763606 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:27:04.763624 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:27:04.767070 (+ 3446us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.161.62:42221: Server oryx/127.27.161.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":259,"threads_started":1}
W20260810 03:27:04.767524 9829 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:27:04.758361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.161.3:60295 (local address 127.27.161.62:42221)
0810 03:27:04.758540 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0810 03:27:04.758544 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:27:04.758846 (+ 302us) server_negotiation.cc:407] Connection header received
0810 03:27:04.758996 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:27:04.759002 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:27:04.759064 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:27:04.759147 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:27:04.759953 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:27:04.760695 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:27:04.762080 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:27:04.762643 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:27:04.762772 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:27:04.767289 (+ 4517us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.161.3:60295: BlockingRecv error: recv got EOF from 127.27.161.3:60295 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Aug 10 03:27:04 dist-test-slave-vpvm krb5kdc[8293](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.161.1@KRBTEST.COM for oryx/127.27.161.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 03:27:04 dist-test-slave-vpvm krb5kdc[8293](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.161.1@KRBTEST.COM for oryx/127.27.161.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 03:27:04.855459 9860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:27:04.845785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:42221 (local address 127.27.161.1:59433)
0810 03:27:04.846300 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0810 03:27:04.846322 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0810 03:27:04.846453 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:27:04.846809 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:27:04.846813 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:27:04.847236 (+ 423us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:27:04.847414 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:27:04.847423 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:27:04.848653 (+ 1230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:27:04.848658 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:27:04.849917 (+ 1259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:27:04.849936 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:27:04.850838 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:27:04.850843 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:27:04.850978 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:27:04.851569 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:27:04.851587 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:27:04.855090 (+ 3503us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.161.62:42221: Server oryx/127.27.161.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":287,"threads_started":1}
W20260810 03:27:04.855625 9829 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:27:04.845984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.161.1:59433 (local address 127.27.161.62:42221)
0810 03:27:04.846221 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0810 03:27:04.846227 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:27:04.846398 (+ 171us) server_negotiation.cc:407] Connection header received
0810 03:27:04.846594 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:27:04.846599 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:27:04.846683 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:27:04.846768 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:27:04.847602 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:27:04.848458 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:27:04.850115 (+ 1657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:27:04.850674 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:27:04.850785 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:27:04.855394 (+ 4609us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.161.1:59433: BlockingRecv error: recv got EOF from 127.27.161.1:59433 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Aug 10 03:27:05 dist-test-slave-vpvm krb5kdc[8293](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.161.2@KRBTEST.COM for oryx/127.27.161.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 03:27:05 dist-test-slave-vpvm krb5kdc[8293](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.161.2@KRBTEST.COM for oryx/127.27.161.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 03:27:05.276027 9861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:27:05.266819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:42221 (local address 127.27.161.2:35621)
0810 03:27:05.267404 (+ 585us) negotiation.cc:107] Waiting for socket to connect
0810 03:27:05.267421 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 03:27:05.267567 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:27:05.267931 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:27:05.267936 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:27:05.268431 (+ 495us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:27:05.268620 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:27:05.268630 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:27:05.269701 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:27:05.269707 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:27:05.270667 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:27:05.270679 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:27:05.271555 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:27:05.271562 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:27:05.271710 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:27:05.272386 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:27:05.272410 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:27:05.275708 (+ 3298us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.161.62:42221: Server oryx/127.27.161.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":340,"threads_started":1}
W20260810 03:27:05.276337 9829 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:27:05.266960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.161.2:35621 (local address 127.27.161.62:42221)
0810 03:27:05.267250 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0810 03:27:05.267256 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:27:05.267513 (+ 257us) server_negotiation.cc:407] Connection header received
0810 03:27:05.267704 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:27:05.267708 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:27:05.267786 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:27:05.267872 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:27:05.268771 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:27:05.269542 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:27:05.270798 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:27:05.271360 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:27:05.271489 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:27:05.275929 (+ 4440us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.161.2:35621: BlockingRecv error: recv got EOF from 127.27.161.2:35621 (error 108)
Metrics: {"server-negotiator.queue_time_us":139}
Aug 10 03:27:05 dist-test-slave-vpvm krb5kdc[8293](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.161.3@KRBTEST.COM for oryx/127.27.161.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 03:27:05 dist-test-slave-vpvm krb5kdc[8293](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.161.3@KRBTEST.COM for oryx/127.27.161.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 03:27:05.777416 9864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:27:05.768329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:42221 (local address 127.27.161.3:35521)
0810 03:27:05.768881 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0810 03:27:05.768899 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 03:27:05.769022 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:27:05.769292 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:27:05.769295 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:27:05.769723 (+ 428us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:27:05.769904 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:27:05.769913 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:27:05.771020 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:27:05.771024 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:27:05.771987 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:27:05.772003 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:27:05.772762 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:27:05.772768 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:27:05.772910 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:27:05.773531 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:27:05.773549 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:27:05.777135 (+ 3586us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.161.62:42221: Server oryx/127.27.161.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":324,"threads_started":1}
W20260810 03:27:05.777626 9829 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:27:05.768389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.161.3:35521 (local address 127.27.161.62:42221)
0810 03:27:05.768588 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0810 03:27:05.768593 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:27:05.768967 (+ 374us) server_negotiation.cc:407] Connection header received
0810 03:27:05.769113 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:27:05.769116 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:27:05.769180 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:27:05.769261 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:27:05.770078 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:27:05.770857 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:27:05.772150 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:27:05.772595 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:27:05.772698 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:27:05.777359 (+ 4661us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.161.3:35521: BlockingRecv error: recv got EOF from 127.27.161.3:35521 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Aug 10 03:27:05 dist-test-slave-vpvm krb5kdc[8293](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.161.1@KRBTEST.COM for oryx/127.27.161.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 03:27:05 dist-test-slave-vpvm krb5kdc[8293](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.161.1@KRBTEST.COM for oryx/127.27.161.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 03:27:05.865777 9865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:27:05.856649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:42221 (local address 127.27.161.1:48863)
0810 03:27:05.857225 (+ 576us) negotiation.cc:107] Waiting for socket to connect
0810 03:27:05.857245 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 03:27:05.857386 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:27:05.857696 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:27:05.857699 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:27:05.858154 (+ 455us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:27:05.858335 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:27:05.858344 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:27:05.859331 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:27:05.859335 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:27:05.860294 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:27:05.860305 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:27:05.860937 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:27:05.860941 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:27:05.861048 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:27:05.861631 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:27:05.861651 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:27:05.865535 (+ 3884us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.161.62:42221: Server oryx/127.27.161.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":249,"threads_started":1}
W20260810 03:27:05.865993 9829 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:27:05.856719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.161.1:48863 (local address 127.27.161.62:42221)
0810 03:27:05.856932 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0810 03:27:05.856940 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 03:27:05.857338 (+ 398us) server_negotiation.cc:407] Connection header received
0810 03:27:05.857501 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:27:05.857504 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:27:05.857581 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:27:05.857654 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:27:05.858458 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:27:05.859201 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:27:05.860406 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:27:05.860825 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:27:05.860914 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:27:05.865773 (+ 4859us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.161.1:48863: BlockingRecv error: recv got EOF from 127.27.161.1:48863 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Aug 10 03:27:06 dist-test-slave-vpvm krb5kdc[8293](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.161.2@KRBTEST.COM for oryx/127.27.161.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 03:27:06 dist-test-slave-vpvm krb5kdc[8293](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.161.2@KRBTEST.COM for oryx/127.27.161.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 03:27:06.286468 9866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:27:06.277155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:42221 (local address 127.27.161.2:48119)
0810 03:27:06.277829 (+ 674us) negotiation.cc:107] Waiting for socket to connect
0810 03:27:06.277845 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 03:27:06.278035 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:27:06.278410 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:27:06.278415 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:27:06.278882 (+ 467us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:27:06.279099 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:27:06.279111 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:27:06.280285 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:27:06.280291 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:27:06.281354 (+ 1063us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:27:06.281369 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:27:06.282064 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:27:06.282069 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:27:06.282194 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:27:06.282751 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:27:06.282770 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:27:06.286202 (+ 3432us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.161.62:42221: Server oryx/127.27.161.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":575,"thread_start_us":290,"threads_started":1}
W20260810 03:27:06.286669 9829 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:27:06.277267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.161.2:48119 (local address 127.27.161.62:42221)
0810 03:27:06.277539 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0810 03:27:06.277546 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:27:06.277942 (+ 396us) server_negotiation.cc:407] Connection header received
0810 03:27:06.278169 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:27:06.278174 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:27:06.278241 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:27:06.278324 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:27:06.279241 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:27:06.280108 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:27:06.281476 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:27:06.281919 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:27:06.282047 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:27:06.286426 (+ 4379us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.161.2:48119: BlockingRecv error: recv got EOF from 127.27.161.2:48119 (error 108)
Metrics: {"server-negotiator.queue_time_us":128}
Aug 10 03:27:06 dist-test-slave-vpvm krb5kdc[8293](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.161.3@KRBTEST.COM for oryx/127.27.161.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 03:27:06 dist-test-slave-vpvm krb5kdc[8293](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.161.3@KRBTEST.COM for oryx/127.27.161.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 03:27:06.788421 9868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:27:06.778779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:42221 (local address 127.27.161.3:57229)
0810 03:27:06.779509 (+ 730us) negotiation.cc:107] Waiting for socket to connect
0810 03:27:06.779527 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 03:27:06.779660 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:27:06.780018 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:27:06.780023 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:27:06.780558 (+ 535us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:27:06.780748 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:27:06.780758 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:27:06.781840 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:27:06.781847 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:27:06.782882 (+ 1035us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:27:06.782898 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:27:06.783729 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:27:06.783734 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:27:06.783875 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:27:06.784529 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:27:06.784549 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:27:06.788158 (+ 3609us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.161.62:42221: Server oryx/127.27.161.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":621,"thread_start_us":392,"threads_started":1}
W20260810 03:27:06.788590 9829 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:27:06.778847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.161.3:57229 (local address 127.27.161.62:42221)
0810 03:27:06.779096 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0810 03:27:06.779103 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:27:06.779614 (+ 511us) server_negotiation.cc:407] Connection header received
0810 03:27:06.779771 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:27:06.779777 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:27:06.779860 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:27:06.779937 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:27:06.780879 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:27:06.781686 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:27:06.783068 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:27:06.783517 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:27:06.783641 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:27:06.788380 (+ 4739us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.161.3:57229: BlockingRecv error: recv got EOF from 127.27.161.3:57229 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Aug 10 03:27:06 dist-test-slave-vpvm krb5kdc[8293](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.161.1@KRBTEST.COM for oryx/127.27.161.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 03:27:06 dist-test-slave-vpvm krb5kdc[8293](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.161.1@KRBTEST.COM for oryx/127.27.161.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 03:27:06.876700 9869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:27:06.866997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:42221 (local address 127.27.161.1:49557)
0810 03:27:06.867691 (+ 694us) negotiation.cc:107] Waiting for socket to connect
0810 03:27:06.867716 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0810 03:27:06.867865 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:27:06.868217 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:27:06.868277 (+ 60us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:27:06.868846 (+ 569us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:27:06.869023 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:27:06.869033 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:27:06.870042 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:27:06.870047 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:27:06.870996 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:27:06.871007 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:27:06.871733 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:27:06.871739 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:27:06.871867 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:27:06.872438 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:27:06.872455 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:27:06.876319 (+ 3864us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.161.62:42221: Server oryx/127.27.161.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":458,"spinlock_wait_cycles":34048,"thread_start_us":408,"threads_started":1}
W20260810 03:27:06.876855 9829 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:27:06.867190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.161.1:49557 (local address 127.27.161.62:42221)
0810 03:27:06.867393 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0810 03:27:06.867399 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:27:06.867793 (+ 394us) server_negotiation.cc:407] Connection header received
0810 03:27:06.867979 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:27:06.867983 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:27:06.868060 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:27:06.868169 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:27:06.869167 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:27:06.869907 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:27:06.871106 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:27:06.871539 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:27:06.871631 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:27:06.876615 (+ 4984us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.161.1:49557: BlockingRecv error: recv got EOF from 127.27.161.1:49557 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Aug 10 03:27:07 dist-test-slave-vpvm krb5kdc[8293](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.161.2@KRBTEST.COM for oryx/127.27.161.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 03:27:07 dist-test-slave-vpvm krb5kdc[8293](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.161.2@KRBTEST.COM for oryx/127.27.161.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 03:27:07.297572 9870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:27:07.287698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:42221 (local address 127.27.161.2:47649)
0810 03:27:07.288340 (+ 642us) negotiation.cc:107] Waiting for socket to connect
0810 03:27:07.288361 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0810 03:27:07.288512 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:27:07.288892 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:27:07.288897 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:27:07.289346 (+ 449us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:27:07.289523 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:27:07.289534 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:27:07.290785 (+ 1251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:27:07.290790 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:27:07.291759 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:27:07.291772 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:27:07.292543 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:27:07.292550 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:27:07.292714 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:27:07.293316 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:27:07.293335 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:27:07.297235 (+ 3900us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.161.62:42221: Server oryx/127.27.161.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":308,"threads_started":1}
W20260810 03:27:07.297731 9829 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:27:07.287948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.161.2:47649 (local address 127.27.161.62:42221)
0810 03:27:07.288314 (+ 366us) server_negotiation.cc:206] Beginning negotiation
0810 03:27:07.288321 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:27:07.288441 (+ 120us) server_negotiation.cc:407] Connection header received
0810 03:27:07.288671 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:27:07.288675 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:27:07.288743 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:27:07.288828 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:27:07.289681 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:27:07.290629 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:27:07.291907 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:27:07.292353 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:27:07.292476 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:27:07.297497 (+ 5021us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.161.2:47649: BlockingRecv error: recv got EOF from 127.27.161.2:47649 (error 108)
Metrics: {"server-negotiator.queue_time_us":213}
Aug 10 03:27:07 dist-test-slave-vpvm krb5kdc[8293](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.161.3@KRBTEST.COM for oryx/127.27.161.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 03:27:07 dist-test-slave-vpvm krb5kdc[8293](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.161.3@KRBTEST.COM for oryx/127.27.161.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 03:27:07.806751 9876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:27:07.789378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:42221 (local address 127.27.161.3:34689)
0810 03:27:07.789914 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0810 03:27:07.789934 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 03:27:07.790087 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:27:07.790361 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:27:07.790364 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:27:07.790793 (+ 429us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:27:07.790961 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:27:07.790970 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:27:07.791992 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:27:07.791995 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:27:07.792942 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:27:07.792955 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:27:07.793603 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:27:07.793607 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:27:07.793718 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:27:07.803214 (+ 9496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:27:07.803248 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:27:07.806492 (+ 3244us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.161.62:42221: Server oryx/127.27.161.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":272,"threads_started":1}
W20260810 03:27:07.806957 9829 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:27:07.789461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.161.3:34689 (local address 127.27.161.62:42221)
0810 03:27:07.789675 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0810 03:27:07.789682 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:27:07.790032 (+ 350us) server_negotiation.cc:407] Connection header received
0810 03:27:07.790185 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:27:07.790189 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:27:07.790254 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:27:07.790330 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:27:07.791108 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:27:07.791882 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:27:07.793050 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:27:07.793467 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:27:07.793556 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:27:07.806733 (+ 13177us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.161.3:34689: BlockingRecv error: recv got EOF from 127.27.161.3:34689 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Aug 10 03:27:07 dist-test-slave-vpvm krb5kdc[8293](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.161.1@KRBTEST.COM for oryx/127.27.161.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 03:27:07 dist-test-slave-vpvm krb5kdc[8293](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.161.1@KRBTEST.COM for oryx/127.27.161.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 03:27:07.887352 9877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:27:07.877866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:42221 (local address 127.27.161.1:57073)
0810 03:27:07.878466 (+ 600us) negotiation.cc:107] Waiting for socket to connect
0810 03:27:07.878504 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0810 03:27:07.878646 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:27:07.878973 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:27:07.878978 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:27:07.879427 (+ 449us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:27:07.879593 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:27:07.879603 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:27:07.880666 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:27:07.880670 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:27:07.881671 (+ 1001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:27:07.881685 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:27:07.882427 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:27:07.882433 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:27:07.882605 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:27:07.883166 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:27:07.883188 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:27:07.887107 (+ 3919us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.161.62:42221: Server oryx/127.27.161.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":359,"threads_started":1}
W20260810 03:27:07.887606 9829 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:27:07.878008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.161.1:57073 (local address 127.27.161.62:42221)
0810 03:27:07.878265 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0810 03:27:07.878273 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 03:27:07.878596 (+ 323us) server_negotiation.cc:407] Connection header received
0810 03:27:07.878767 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:27:07.878772 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:27:07.878837 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:27:07.878920 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:27:07.879742 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:27:07.880529 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:27:07.881806 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:27:07.882272 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:27:07.882385 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:27:07.887336 (+ 4951us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.161.1:57073: BlockingRecv error: recv got EOF from 127.27.161.1:57073 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
Aug 10 03:27:08 dist-test-slave-vpvm krb5kdc[8293](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.161.2@KRBTEST.COM for oryx/127.27.161.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 03:27:08 dist-test-slave-vpvm krb5kdc[8293](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.161.2@KRBTEST.COM for oryx/127.27.161.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 03:27:08.307559 9879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:27:08.298782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:42221 (local address 127.27.161.2:35099)
0810 03:27:08.299349 (+ 567us) negotiation.cc:107] Waiting for socket to connect
0810 03:27:08.299365 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 03:27:08.299492 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:27:08.299795 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:27:08.299798 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:27:08.300204 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:27:08.300357 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:27:08.300366 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:27:08.301326 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:27:08.301330 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:27:08.302325 (+ 995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:27:08.302339 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:27:08.303027 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:27:08.303031 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:27:08.303170 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:27:08.303771 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:27:08.303790 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:27:08.307315 (+ 3525us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.161.62:42221: Server oryx/127.27.161.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":285,"threads_started":1}
W20260810 03:27:08.307819 9829 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:27:08.298889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.161.2:35099 (local address 127.27.161.62:42221)
0810 03:27:08.299094 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0810 03:27:08.299102 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 03:27:08.299441 (+ 339us) server_negotiation.cc:407] Connection header received
0810 03:27:08.299604 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:27:08.299608 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:27:08.299671 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:27:08.299753 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:27:08.300465 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:27:08.301219 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:27:08.302445 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:27:08.302904 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:27:08.302999 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:27:08.307543 (+ 4544us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.161.2:35099: BlockingRecv error: recv got EOF from 127.27.161.2:35099 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Aug 10 03:27:08 dist-test-slave-vpvm krb5kdc[8293](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.161.3@KRBTEST.COM for oryx/127.27.161.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 03:27:08 dist-test-slave-vpvm krb5kdc[8293](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.161.3@KRBTEST.COM for oryx/127.27.161.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 03:27:08.817508 9880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:27:08.807886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:42221 (local address 127.27.161.3:46585)
0810 03:27:08.808525 (+ 639us) negotiation.cc:107] Waiting for socket to connect
0810 03:27:08.808549 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0810 03:27:08.808707 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:27:08.809016 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:27:08.809019 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:27:08.809409 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:27:08.809561 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:27:08.809571 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:27:08.810563 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:27:08.810567 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:27:08.811524 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:27:08.811537 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:27:08.812398 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:27:08.812403 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:27:08.812548 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:27:08.813270 (+ 722us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:27:08.813289 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:27:08.817260 (+ 3971us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.161.62:42221: Server oryx/127.27.161.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":502,"thread_start_us":326,"threads_started":1}
W20260810 03:27:08.817720 9829 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:27:08.808044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.161.3:46585 (local address 127.27.161.62:42221)
0810 03:27:08.808309 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0810 03:27:08.808314 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:27:08.808663 (+ 349us) server_negotiation.cc:407] Connection header received
0810 03:27:08.808812 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:27:08.808816 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:27:08.808883 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:27:08.808967 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:27:08.809700 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:27:08.810454 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:27:08.811695 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:27:08.812155 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:27:08.812277 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:27:08.817507 (+ 5230us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.161.3:46585: BlockingRecv error: recv got EOF from 127.27.161.3:46585 (error 108)
Metrics: {"server-negotiator.queue_time_us":111}
Aug 10 03:27:08 dist-test-slave-vpvm krb5kdc[8293](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.161.1@KRBTEST.COM for oryx/127.27.161.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 03:27:08 dist-test-slave-vpvm krb5kdc[8293](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.161.1@KRBTEST.COM for oryx/127.27.161.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 03:27:08.897696 9881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:27:08.888507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:42221 (local address 127.27.161.1:51177)
0810 03:27:08.889037 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0810 03:27:08.889076 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0810 03:27:08.889223 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:27:08.889539 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:27:08.889544 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:27:08.889932 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:27:08.890149 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:27:08.890160 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:27:08.891154 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:27:08.891158 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:27:08.892229 (+ 1071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:27:08.892243 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:27:08.892950 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:27:08.892956 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:27:08.893126 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:27:08.893769 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:27:08.893788 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:27:08.897434 (+ 3646us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.161.62:42221: Server oryx/127.27.161.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":420,"threads_started":1}
W20260810 03:27:08.898002 9829 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:27:08.888684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.161.1:51177 (local address 127.27.161.62:42221)
0810 03:27:08.888943 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0810 03:27:08.888950 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:27:08.889159 (+ 209us) server_negotiation.cc:407] Connection header received
0810 03:27:08.889343 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:27:08.889347 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:27:08.889408 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:27:08.889486 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:27:08.890294 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:27:08.891021 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:27:08.892378 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:27:08.892802 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:27:08.892912 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:27:08.897713 (+ 4801us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.161.1:51177: BlockingRecv error: recv got EOF from 127.27.161.1:51177 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Aug 10 03:27:09 dist-test-slave-vpvm krb5kdc[8293](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.161.2@KRBTEST.COM for oryx/127.27.161.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 03:27:09 dist-test-slave-vpvm krb5kdc[8293](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.161.2@KRBTEST.COM for oryx/127.27.161.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 03:27:09.318440 9883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:27:09.308689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:42221 (local address 127.27.161.2:53251)
0810 03:27:09.309306 (+ 617us) negotiation.cc:107] Waiting for socket to connect
0810 03:27:09.309324 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 03:27:09.309513 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:27:09.309894 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:27:09.309899 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:27:09.310389 (+ 490us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:27:09.310596 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:27:09.310608 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:27:09.311741 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:27:09.311747 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:27:09.312820 (+ 1073us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:27:09.312834 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:27:09.313579 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:27:09.313586 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:27:09.313774 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:27:09.314397 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:27:09.314418 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:27:09.318081 (+ 3663us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.161.62:42221: Server oryx/127.27.161.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":312,"threads_started":1}
W20260810 03:27:09.318565 9829 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:27:09.308869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.161.2:53251 (local address 127.27.161.62:42221)
0810 03:27:09.309158 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0810 03:27:09.309165 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:27:09.309417 (+ 252us) server_negotiation.cc:407] Connection header received
0810 03:27:09.309666 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:27:09.309670 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:27:09.309744 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:27:09.309840 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:27:09.310770 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:27:09.311584 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:27:09.312987 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:27:09.313407 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:27:09.313535 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:27:09.318316 (+ 4781us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.161.2:53251: BlockingRecv error: recv got EOF from 127.27.161.2:53251 (error 108)
Metrics: {"server-negotiator.queue_time_us":107}
Aug 10 03:27:09 dist-test-slave-vpvm krb5kdc[8293](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.161.3@KRBTEST.COM for oryx/127.27.161.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 03:27:09 dist-test-slave-vpvm krb5kdc[8293](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.161.3@KRBTEST.COM for oryx/127.27.161.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 03:27:09.827824 9884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:27:09.818580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:42221 (local address 127.27.161.3:46299)
0810 03:27:09.819392 (+ 812us) negotiation.cc:107] Waiting for socket to connect
0810 03:27:09.819410 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 03:27:09.819564 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:27:09.819890 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:27:09.819895 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:27:09.820294 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:27:09.820467 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:27:09.820479 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:27:09.821567 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:27:09.821571 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:27:09.822630 (+ 1059us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:27:09.822646 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:27:09.823405 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:27:09.823411 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:27:09.823587 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:27:09.824208 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:27:09.824227 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:27:09.827567 (+ 3340us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.161.62:42221: Server oryx/127.27.161.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":702,"thread_start_us":575,"threads_started":1}
W20260810 03:27:09.828020 9829 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:27:09.818793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.161.3:46299 (local address 127.27.161.62:42221)
0810 03:27:09.819075 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0810 03:27:09.819081 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:27:09.819501 (+ 420us) server_negotiation.cc:407] Connection header received
0810 03:27:09.819685 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:27:09.819691 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:27:09.819756 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:27:09.819838 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:27:09.820638 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:27:09.821409 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:27:09.822774 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:27:09.823246 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:27:09.823362 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:27:09.827780 (+ 4418us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.161.3:46299: BlockingRecv error: recv got EOF from 127.27.161.3:46299 (error 108)
Metrics: {"server-negotiator.queue_time_us":137}
Aug 10 03:27:09 dist-test-slave-vpvm krb5kdc[8293](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.161.1@KRBTEST.COM for oryx/127.27.161.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 03:27:09 dist-test-slave-vpvm krb5kdc[8293](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.161.1@KRBTEST.COM for oryx/127.27.161.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 03:27:09.909123 9885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:27:09.898745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:42221 (local address 127.27.161.1:50495)
0810 03:27:09.899271 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0810 03:27:09.899296 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0810 03:27:09.899422 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:27:09.899814 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:27:09.899818 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:27:09.900266 (+ 448us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:27:09.900453 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:27:09.900463 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:27:09.901437 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:27:09.901440 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:27:09.902453 (+ 1013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:27:09.902465 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:27:09.903216 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:27:09.903232 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:27:09.903358 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:27:09.903937 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:27:09.903958 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:27:09.908848 (+ 4890us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.161.62:42221: Server oryx/127.27.161.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":301,"threads_started":1}
W20260810 03:27:09.909336 9829 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:27:09.898940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.161.1:50495 (local address 127.27.161.62:42221)
0810 03:27:09.899144 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0810 03:27:09.899149 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:27:09.899369 (+ 220us) server_negotiation.cc:407] Connection header received
0810 03:27:09.899590 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:27:09.899595 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:27:09.899664 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:27:09.899755 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:27:09.900602 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:27:09.901331 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:27:09.902616 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:27:09.903090 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:27:09.903196 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:27:09.909113 (+ 5917us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.161.1:50495: BlockingRecv error: recv got EOF from 127.27.161.1:50495 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Aug 10 03:27:10 dist-test-slave-vpvm krb5kdc[8293](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.161.2@KRBTEST.COM for oryx/127.27.161.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 03:27:10 dist-test-slave-vpvm krb5kdc[8293](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.161.2@KRBTEST.COM for oryx/127.27.161.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 03:27:10.328730 9887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:27:10.319473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:42221 (local address 127.27.161.2:37133)
0810 03:27:10.320218 (+ 745us) negotiation.cc:107] Waiting for socket to connect
0810 03:27:10.320236 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 03:27:10.320375 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:27:10.320721 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:27:10.320725 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:27:10.321202 (+ 477us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:27:10.321385 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:27:10.321395 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:27:10.322439 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:27:10.322443 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:27:10.323451 (+ 1008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:27:10.323467 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:27:10.324083 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:27:10.324087 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:27:10.324214 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:27:10.324789 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:27:10.324806 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:27:10.328355 (+ 3549us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.161.62:42221: Server oryx/127.27.161.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":634,"thread_start_us":507,"threads_started":1}
W20260810 03:27:10.328910 9829 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:27:10.319624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.161.2:37133 (local address 127.27.161.62:42221)
0810 03:27:10.319847 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0810 03:27:10.319855 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 03:27:10.320326 (+ 471us) server_negotiation.cc:407] Connection header received
0810 03:27:10.320492 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:27:10.320496 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:27:10.320572 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:27:10.320680 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:27:10.321511 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:27:10.322344 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:27:10.323577 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:27:10.323965 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:27:10.324055 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:27:10.328630 (+ 4575us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.161.2:37133: BlockingRecv error: recv got EOF from 127.27.161.2:37133 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Aug 10 03:27:10 dist-test-slave-vpvm krb5kdc[8293](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.161.3@KRBTEST.COM for oryx/127.27.161.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 03:27:10 dist-test-slave-vpvm krb5kdc[8293](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.161.3@KRBTEST.COM for oryx/127.27.161.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 03:27:10.838698 9888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:27:10.828876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:42221 (local address 127.27.161.3:49097)
0810 03:27:10.829389 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0810 03:27:10.829407 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 03:27:10.829545 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:27:10.829837 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:27:10.829840 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:27:10.830311 (+ 471us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:27:10.830493 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:27:10.830503 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:27:10.831578 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:27:10.831582 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:27:10.832504 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:27:10.832517 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:27:10.833312 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:27:10.833318 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:27:10.833466 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:27:10.834144 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:27:10.834163 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:27:10.838383 (+ 4220us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.161.62:42221: Server oryx/127.27.161.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":284,"threads_started":1}
W20260810 03:27:10.838837 9829 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:27:10.829054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.161.3:49097 (local address 127.27.161.62:42221)
0810 03:27:10.829273 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0810 03:27:10.829279 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:27:10.829519 (+ 240us) server_negotiation.cc:407] Connection header received
0810 03:27:10.829646 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:27:10.829649 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:27:10.829717 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:27:10.829806 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:27:10.830666 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:27:10.831429 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:27:10.832655 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:27:10.833135 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:27:10.833235 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:27:10.838617 (+ 5382us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.161.3:49097: BlockingRecv error: recv got EOF from 127.27.161.3:49097 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
Aug 10 03:27:10 dist-test-slave-vpvm krb5kdc[8293](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.161.1@KRBTEST.COM for oryx/127.27.161.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 03:27:10 dist-test-slave-vpvm krb5kdc[8293](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.161.1@KRBTEST.COM for oryx/127.27.161.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 03:27:10.919569 9889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:27:10.910330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:42221 (local address 127.27.161.1:43523)
0810 03:27:10.910919 (+ 589us) negotiation.cc:107] Waiting for socket to connect
0810 03:27:10.910946 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0810 03:27:10.911083 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:27:10.911465 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:27:10.911469 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:27:10.911941 (+ 472us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:27:10.912123 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:27:10.912134 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:27:10.913188 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:27:10.913194 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:27:10.914154 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:27:10.914166 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:27:10.914940 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:27:10.914946 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:27:10.915117 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:27:10.915720 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:27:10.915738 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:27:10.919187 (+ 3449us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.161.62:42221: Server oryx/127.27.161.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":388,"threads_started":1}
W20260810 03:27:10.919703 9829 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:27:10.910463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.161.1:43523 (local address 127.27.161.62:42221)
0810 03:27:10.910704 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0810 03:27:10.910711 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:27:10.911036 (+ 325us) server_negotiation.cc:407] Connection header received
0810 03:27:10.911207 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:27:10.911211 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:27:10.911291 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:27:10.911390 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:27:10.912285 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:27:10.913040 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:27:10.914291 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:27:10.914759 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:27:10.914871 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:27:10.919426 (+ 4555us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.161.1:43523: BlockingRecv error: recv got EOF from 127.27.161.1:43523 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
Aug 10 03:27:11 dist-test-slave-vpvm krb5kdc[8293](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.161.2@KRBTEST.COM for oryx/127.27.161.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 03:27:11 dist-test-slave-vpvm krb5kdc[8293](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.161.2@KRBTEST.COM for oryx/127.27.161.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 03:27:11.339537 9891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:27:11.329817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:42221 (local address 127.27.161.2:57941)
0810 03:27:11.330454 (+ 637us) negotiation.cc:107] Waiting for socket to connect
0810 03:27:11.330471 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 03:27:11.330597 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:27:11.330894 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:27:11.330897 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:27:11.331334 (+ 437us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:27:11.331514 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:27:11.331524 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:27:11.332743 (+ 1219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:27:11.332748 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:27:11.333746 (+ 998us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:27:11.333762 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:27:11.334660 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:27:11.334666 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:27:11.334817 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:27:11.335469 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:27:11.335491 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:27:11.339266 (+ 3775us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.161.62:42221: Server oryx/127.27.161.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":530,"thread_start_us":354,"threads_started":1}
W20260810 03:27:11.339766 9829 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:27:11.329844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.161.2:57941 (local address 127.27.161.62:42221)
0810 03:27:11.330074 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0810 03:27:11.330079 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:27:11.330592 (+ 513us) server_negotiation.cc:407] Connection header received
0810 03:27:11.330697 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:27:11.330701 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:27:11.330768 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:27:11.330854 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:27:11.331669 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:27:11.332436 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:27:11.333976 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:27:11.334424 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:27:11.334539 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:27:11.339509 (+ 4970us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.161.2:57941: BlockingRecv error: recv got EOF from 127.27.161.2:57941 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Aug 10 03:27:11 dist-test-slave-vpvm krb5kdc[8293](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.161.3@KRBTEST.COM for oryx/127.27.161.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 03:27:11 dist-test-slave-vpvm krb5kdc[8293](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.161.3@KRBTEST.COM for oryx/127.27.161.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 03:27:11.850653 9893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:27:11.840025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:42221 (local address 127.27.161.3:56049)
0810 03:27:11.840632 (+ 607us) negotiation.cc:107] Waiting for socket to connect
0810 03:27:11.840655 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0810 03:27:11.840813 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:27:11.841186 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:27:11.841192 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:27:11.841784 (+ 592us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:27:11.842107 (+ 323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:27:11.842123 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:27:11.843129 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:27:11.843135 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:27:11.844728 (+ 1593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:27:11.844746 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:27:11.845494 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:27:11.845500 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:27:11.845682 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:27:11.846305 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:27:11.846323 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:27:11.850352 (+ 4029us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.161.62:42221: Server oryx/127.27.161.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":274,"threads_started":1}
W20260810 03:27:11.850908 9829 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:27:11.840041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.161.3:56049 (local address 127.27.161.62:42221)
0810 03:27:11.840271 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0810 03:27:11.840278 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:27:11.840745 (+ 467us) server_negotiation.cc:407] Connection header received
0810 03:27:11.840950 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:27:11.840954 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:27:11.841039 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:27:11.841142 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:27:11.842244 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:27:11.842980 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:27:11.844870 (+ 1890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:27:11.845280 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:27:11.845398 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:27:11.850628 (+ 5230us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.161.3:56049: BlockingRecv error: recv got EOF from 127.27.161.3:56049 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Aug 10 03:27:11 dist-test-slave-vpvm krb5kdc[8293](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.161.1@KRBTEST.COM for oryx/127.27.161.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 03:27:11 dist-test-slave-vpvm krb5kdc[8293](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.161.1@KRBTEST.COM for oryx/127.27.161.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 03:27:11.931502 9894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:27:11.920645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:42221 (local address 127.27.161.1:47759)
0810 03:27:11.921120 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0810 03:27:11.921142 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0810 03:27:11.921264 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:27:11.921593 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:27:11.921596 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:27:11.922046 (+ 450us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:27:11.922215 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:27:11.922225 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:27:11.923797 (+ 1572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:27:11.923802 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:27:11.924803 (+ 1001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:27:11.924821 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:27:11.925812 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:27:11.925817 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:27:11.925984 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:27:11.926787 (+ 803us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:27:11.926806 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:27:11.931122 (+ 4316us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.161.62:42221: Server oryx/127.27.161.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":246,"threads_started":1}
W20260810 03:27:11.931546 9829 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:27:11.920784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.161.1:47759 (local address 127.27.161.62:42221)
0810 03:27:11.921014 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0810 03:27:11.921041 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0810 03:27:11.921223 (+ 182us) server_negotiation.cc:407] Connection header received
0810 03:27:11.921373 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:27:11.921378 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:27:11.921464 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:27:11.921580 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:27:11.922366 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:27:11.923591 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:27:11.924967 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:27:11.925649 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:27:11.925789 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:27:11.931343 (+ 5554us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.161.1:47759: BlockingRecv error: recv got EOF from 127.27.161.1:47759 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Aug 10 03:27:12 dist-test-slave-vpvm krb5kdc[8293](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.161.2@KRBTEST.COM for oryx/127.27.161.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 03:27:12 dist-test-slave-vpvm krb5kdc[8293](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.161.2@KRBTEST.COM for oryx/127.27.161.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 03:27:12.350638 9895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:27:12.340801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:42221 (local address 127.27.161.2:41749)
0810 03:27:12.341320 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0810 03:27:12.341341 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0810 03:27:12.341490 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:27:12.341817 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:27:12.341821 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:27:12.342337 (+ 516us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:27:12.342497 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:27:12.342508 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:27:12.343620 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:27:12.343625 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:27:12.344580 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:27:12.344593 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:27:12.345388 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:27:12.345394 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:27:12.345577 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:27:12.346333 (+ 756us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:27:12.346360 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:27:12.350301 (+ 3941us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.161.62:42221: Server oryx/127.27.161.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":251,"threads_started":1}
W20260810 03:27:12.350843 9829 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:27:12.341089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.161.2:41749 (local address 127.27.161.62:42221)
0810 03:27:12.341322 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0810 03:27:12.341328 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:27:12.341425 (+ 97us) server_negotiation.cc:407] Connection header received
0810 03:27:12.341615 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:27:12.341619 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:27:12.341683 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:27:12.341767 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:27:12.342661 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:27:12.343458 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:27:12.344765 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:27:12.345220 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:27:12.345342 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:27:12.350569 (+ 5227us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.161.2:41749: BlockingRecv error: recv got EOF from 127.27.161.2:41749 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
Aug 10 03:27:12 dist-test-slave-vpvm krb5kdc[8293](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.161.3@KRBTEST.COM for oryx/127.27.161.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 03:27:12 dist-test-slave-vpvm krb5kdc[8293](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.161.3@KRBTEST.COM for oryx/127.27.161.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 03:27:12.865103 9896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:27:12.851876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:42221 (local address 127.27.161.3:34855)
0810 03:27:12.852373 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0810 03:27:12.852391 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 03:27:12.852515 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:27:12.852797 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:27:12.852800 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:27:12.853209 (+ 409us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:27:12.853366 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:27:12.853376 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:27:12.854455 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:27:12.854459 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:27:12.855459 (+ 1000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:27:12.855475 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:27:12.856169 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:27:12.856173 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:27:12.856295 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:27:12.861165 (+ 4870us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:27:12.861193 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:27:12.864837 (+ 3644us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.161.62:42221: Server oryx/127.27.161.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":258,"threads_started":1}
W20260810 03:27:12.865311 9898 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:27:12.851919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.161.3:34855 (local address 127.27.161.62:42221)
0810 03:27:12.852390 (+ 471us) server_negotiation.cc:206] Beginning negotiation
0810 03:27:12.852394 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:27:12.852462 (+ 68us) server_negotiation.cc:407] Connection header received
0810 03:27:12.852620 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:27:12.852624 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:27:12.852687 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:27:12.852768 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:27:12.853480 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:27:12.854339 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:27:12.855579 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:27:12.856058 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:27:12.856160 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:27:12.865070 (+ 8910us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.161.3:34855: BlockingRecv error: recv got EOF from 127.27.161.3:34855 (error 108)
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":246,"threads_started":1}
Aug 10 03:27:12 dist-test-slave-vpvm krb5kdc[8293](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.161.1@KRBTEST.COM for oryx/127.27.161.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 03:27:12 dist-test-slave-vpvm krb5kdc[8293](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.161.1@KRBTEST.COM for oryx/127.27.161.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 03:27:12.942282 9899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:27:12.933090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:42221 (local address 127.27.161.1:50033)
0810 03:27:12.933712 (+ 622us) negotiation.cc:107] Waiting for socket to connect
0810 03:27:12.933735 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0810 03:27:12.933877 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:27:12.934293 (+ 416us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:27:12.934299 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:27:12.934706 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:27:12.934902 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:27:12.934912 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:27:12.936113 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:27:12.936117 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:27:12.937053 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:27:12.937065 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:27:12.937809 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:27:12.937814 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:27:12.937945 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:27:12.938539 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:27:12.938559 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:27:12.941878 (+ 3319us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.161.62:42221: Server oryx/127.27.161.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":292,"threads_started":1}
W20260810 03:27:12.942413 9898 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:27:12.933213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.161.1:50033 (local address 127.27.161.62:42221)
0810 03:27:12.933584 (+ 371us) server_negotiation.cc:206] Beginning negotiation
0810 03:27:12.933592 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 03:27:12.933831 (+ 239us) server_negotiation.cc:407] Connection header received
0810 03:27:12.934055 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:27:12.934059 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:27:12.934133 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:27:12.934228 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:27:12.935046 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:27:12.935941 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:27:12.937212 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:27:12.937665 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:27:12.937764 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:27:12.942152 (+ 4388us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.161.1:50033: BlockingRecv error: recv got EOF from 127.27.161.1:50033 (error 108)
Metrics: {"server-negotiator.queue_time_us":211}
Aug 10 03:27:13 dist-test-slave-vpvm krb5kdc[8293](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.161.2@KRBTEST.COM for oryx/127.27.161.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 03:27:13 dist-test-slave-vpvm krb5kdc[8293](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.161.2@KRBTEST.COM for oryx/127.27.161.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 03:27:13.361801 9900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:27:13.352269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:42221 (local address 127.27.161.2:54907)
0810 03:27:13.352965 (+ 696us) negotiation.cc:107] Waiting for socket to connect
0810 03:27:13.353002 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0810 03:27:13.353148 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:27:13.353531 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:27:13.353536 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:27:13.353944 (+ 408us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:27:13.354202 (+ 258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:27:13.354213 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:27:13.355266 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:27:13.355271 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:27:13.356334 (+ 1063us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:27:13.356349 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:27:13.357177 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:27:13.357183 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:27:13.357334 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:27:13.357935 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:27:13.357983 (+ 48us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:27:13.361506 (+ 3523us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.161.62:42221: Server oryx/127.27.161.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":545,"thread_start_us":416,"threads_started":1}
W20260810 03:27:13.362046 9898 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:27:13.352403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.161.2:54907 (local address 127.27.161.62:42221)
0810 03:27:13.352614 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0810 03:27:13.352621 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:27:13.353108 (+ 487us) server_negotiation.cc:407] Connection header received
0810 03:27:13.353269 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:27:13.353274 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:27:13.353365 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:27:13.353451 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:27:13.354343 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:27:13.355113 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:27:13.356471 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:27:13.357002 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:27:13.357111 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:27:13.361772 (+ 4661us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.161.2:54907: BlockingRecv error: recv got EOF from 127.27.161.2:54907 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
I20260810 03:27:13.638454 28292 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iBH5/build/asan/bin/kudu pid 9174
I20260810 03:27:13.652135 9902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:27:13.639830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34402 (local address 127.27.161.1:46369)
0810 03:27:13.640465 (+ 635us) server_negotiation.cc:206] Beginning negotiation
0810 03:27:13.640472 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:27:13.640499 (+ 27us) server_negotiation.cc:407] Connection header received
0810 03:27:13.640594 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:27:13.640601 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:27:13.640830 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:27:13.641009 (+ 179us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:27:13.642702 (+ 1693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:27:13.643857 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:27:13.645333 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:27:13.645977 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:27:13.646095 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:27:13.648764 (+ 2669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:27:13.648774 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:27:13.648791 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:27:13.648852 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:27:13.651585 (+ 2733us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.27.161.1@KRBTEST.COM not found in keytab (ticket kvno 2)
0810 03:27:13.651907 (+ 322us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:34402: Request ticket server kudu/127.27.161.1@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":466,"thread_start_us":318,"threads_started":1}
W20260810 03:27:13.652240 9902 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:34402: Request ticket server kudu/127.27.161.1@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260810 03:27:13.652328 28292 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task24iBH5/build/asan/bin/kudu pid 9174: Not authorized: Client connection negotiation failed: client connection to 127.27.161.1:46369: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.27.161.1@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260810 03:27:13.652590 28292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iBH5/build/asan/bin/kudu with pid 9174
I20260810 03:27:13.683759 28292 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iBH5/build/asan/bin/kudu pid 9395
I20260810 03:27:13.697069 9903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:27:13.684975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47892 (local address 127.27.161.2:39655)
0810 03:27:13.685620 (+ 645us) server_negotiation.cc:206] Beginning negotiation
0810 03:27:13.685628 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 03:27:13.685661 (+ 33us) server_negotiation.cc:407] Connection header received
0810 03:27:13.685733 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:27:13.685740 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:27:13.686054 (+ 314us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:27:13.686266 (+ 212us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:27:13.687677 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:27:13.688816 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:27:13.690107 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:27:13.690731 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:27:13.690858 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:27:13.693559 (+ 2701us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:27:13.693569 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:27:13.693586 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:27:13.693636 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:27:13.696469 (+ 2833us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.27.161.2@KRBTEST.COM not found in keytab (ticket kvno 2)
0810 03:27:13.696816 (+ 347us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:47892: Request ticket server kudu/127.27.161.2@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":447,"thread_start_us":278,"threads_started":1}
W20260810 03:27:13.697199 9903 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:47892: Request ticket server kudu/127.27.161.2@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260810 03:27:13.697289 28292 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task24iBH5/build/asan/bin/kudu pid 9395: Not authorized: Client connection negotiation failed: client connection to 127.27.161.2:39655: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.27.161.2@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260810 03:27:13.697484 28292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iBH5/build/asan/bin/kudu with pid 9395
I20260810 03:27:13.721897 28292 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iBH5/build/asan/bin/kudu pid 9605
I20260810 03:27:13.735986 9904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:27:13.723606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36804 (local address 127.27.161.3:35967)
0810 03:27:13.724372 (+ 766us) server_negotiation.cc:206] Beginning negotiation
0810 03:27:13.724381 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 03:27:13.724418 (+ 37us) server_negotiation.cc:407] Connection header received
0810 03:27:13.724501 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:27:13.724510 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:27:13.724785 (+ 275us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:27:13.725038 (+ 253us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:27:13.726663 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:27:13.727734 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:27:13.729047 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:27:13.729667 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:27:13.729804 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:27:13.732594 (+ 2790us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:27:13.732605 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:27:13.732623 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:27:13.732679 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:27:13.735269 (+ 2590us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.27.161.3@KRBTEST.COM not found in keytab (ticket kvno 2)
0810 03:27:13.735651 (+ 382us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:36804: Request ticket server kudu/127.27.161.3@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":557,"thread_start_us":400,"threads_started":1}
W20260810 03:27:13.736114 9904 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:36804: Request ticket server kudu/127.27.161.3@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260810 03:27:13.736163 28292 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task24iBH5/build/asan/bin/kudu pid 9605: Not authorized: Client connection negotiation failed: client connection to 127.27.161.3:35967: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.27.161.3@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260810 03:27:13.736326 28292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iBH5/build/asan/bin/kudu with pid 9605
I20260810 03:27:13.763947 28292 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iBH5/build/asan/bin/kudu pid 9075
I20260810 03:27:13.765509 9164 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:27:13.888712 28292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iBH5/build/asan/bin/kudu with pid 9075
2026-08-10T03:27:13Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (21840 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-08-10T03:27:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T03:27:13Z Disabled control of system clock
I20260810 03:27:13.992859 28292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iBH5/build/asan/bin/kudu
/tmp/dist-test-task24iBH5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786332336319544-28292-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786332336319544-28292-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786332336319544-28292-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786332336319544-28292-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.161.62:40703
--webserver_interface=127.27.161.62
--webserver_port=0
--builtin_ntp_servers=127.27.161.20:39911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.161.62:40703
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260810 03:27:14.362845 9913 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:27:14.363341 9913 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:27:14.363482 9913 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:27:14.373239 9913 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:27:14.373365 9913 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:27:14.373404 9913 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:27:14.373462 9913 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:27:14.373505 9913 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:27:14.383160 9913 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.161.20:39911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786332336319544-28292-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786332336319544-28292-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.161.62:40703
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.161.62:40703
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786332336319544-28292-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.161.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786332336319544-28292-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:27:14.384557 9913 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:27:14.386520 9913 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:27:14.399123 9925 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:27:14.399108 9928 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:27:14.399163 9924 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:27:14.399601 9913 server_base.cc:1034] running on GCE node
I20260810 03:27:14.400686 9913 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:27:14.402212 9913 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:27:14.403419 9913 hybrid_clock.cc:648] HybridClock initialized: now 1786332434403373 us; error 30 us; skew 500 ppm
I20260810 03:27:14.403785 9913 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 03:27:14.406272 9913 webserver.cc:460] Webserver started at http://127.27.161.62:39755/ using document root <none> and password file <none>
I20260810 03:27:14.407007 9913 fs_manager.cc:359] Metadata directory not provided
I20260810 03:27:14.407104 9913 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:27:14.407438 9913 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:27:14.410542 9913 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786332336319544-28292-0/minicluster-data/master-0/data/instance:
uuid: "07cb12b3ca0b492bbfa9c463582d2e3a"
format_stamp: "Formatted at 2026-08-10 03:27:14 on dist-test-slave-vpvm"
server_key: "cffd8ab58f6583b3b6b5a25847cedc59"
server_key_iv: "8f555007291451d5bc3c8bca689110a8"
server_key_version: "encryptionkey@0"
I20260810 03:27:14.411356 9913 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786332336319544-28292-0/minicluster-data/master-0/wal/instance:
uuid: "07cb12b3ca0b492bbfa9c463582d2e3a"
format_stamp: "Formatted at 2026-08-10 03:27:14 on dist-test-slave-vpvm"
server_key: "cffd8ab58f6583b3b6b5a25847cedc59"
server_key_iv: "8f555007291451d5bc3c8bca689110a8"
server_key_version: "encryptionkey@0"
I20260810 03:27:14.416839 9913 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260810 03:27:14.420707 9936 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:27:14.422248 9913 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 03:27:14.422446 9913 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786332336319544-28292-0/minicluster-data/master-0/data,/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786332336319544-28292-0/minicluster-data/master-0/wal
uuid: "07cb12b3ca0b492bbfa9c463582d2e3a"
format_stamp: "Formatted at 2026-08-10 03:27:14 on dist-test-slave-vpvm"
server_key: "cffd8ab58f6583b3b6b5a25847cedc59"
server_key_iv: "8f555007291451d5bc3c8bca689110a8"
server_key_version: "encryptionkey@0"
I20260810 03:27:14.422686 9913 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786332336319544-28292-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786332336319544-28292-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786332336319544-28292-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:27:14.436262 9913 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:27:14.437209 9913 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:27:14.437521 9913 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:27:14.456488 9913 rpc_server.cc:307] RPC server started. Bound to: 127.27.161.62:40703
I20260810 03:27:14.456532 10003 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.161.62:40703 every 8 connection(s)
I20260810 03:27:14.457931 9913 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786332336319544-28292-0/minicluster-data/master-0/data/info.pb
I20260810 03:27:14.461985 10005 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:27:14.464012 28292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iBH5/build/asan/bin/kudu as pid 9913
I20260810 03:27:14.464232 28292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786332336319544-28292-0/minicluster-data/master-0/wal/instance
I20260810 03:27:14.464876 28292 external_mini_cluster.cc:1428] Setting key e5d7a09fa54fa9999c9f88726de4f673
I20260810 03:27:14.476179 10005 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 07cb12b3ca0b492bbfa9c463582d2e3a: Bootstrap starting.
I20260810 03:27:14.479707 10005 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 07cb12b3ca0b492bbfa9c463582d2e3a: Neither blocks nor log segments found. Creating new log.
I20260810 03:27:14.480911 10005 log.cc:824] T 00000000000000000000000000000000 P 07cb12b3ca0b492bbfa9c463582d2e3a: Log is configured to *not* fsync() on all Append() calls
I20260810 03:27:14.481189 10010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:27:14.467413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50818 (local address 127.27.161.62:40703)
0810 03:27:14.468711 (+ 1298us) server_negotiation.cc:206] Beginning negotiation
0810 03:27:14.468728 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0810 03:27:14.468785 (+ 57us) server_negotiation.cc:407] Connection header received
0810 03:27:14.471178 (+ 2393us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:27:14.471213 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:27:14.471879 (+ 666us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:27:14.472381 (+ 502us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:27:14.472970 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:27:14.474997 (+ 2027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:27:14.476460 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:27:14.477205 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:27:14.477391 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:27:14.478717 (+ 1326us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:27:14.478726 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:27:14.478741 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:27:14.479300 (+ 559us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:27:14.479397 (+ 97us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:27:14.479421 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:27:14.479537 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0810 03:27:14.479774 (+ 237us) server_negotiation.cc:299] Negotiation successful
0810 03:27:14.480106 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":695,"thread_start_us":385,"threads_started":1}
I20260810 03:27:14.484391 10005 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 07cb12b3ca0b492bbfa9c463582d2e3a: No bootstrap required, opened a new log
I20260810 03:27:14.492213 10005 raft_consensus.cc:348] T 00000000000000000000000000000000 P 07cb12b3ca0b492bbfa9c463582d2e3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07cb12b3ca0b492bbfa9c463582d2e3a" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 40703 } }
I20260810 03:27:14.492622 10005 raft_consensus.cc:374] T 00000000000000000000000000000000 P 07cb12b3ca0b492bbfa9c463582d2e3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:27:14.492726 10005 raft_consensus.cc:729] T 00000000000000000000000000000000 P 07cb12b3ca0b492bbfa9c463582d2e3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07cb12b3ca0b492bbfa9c463582d2e3a, State: Initialized, Role: FOLLOWER
I20260810 03:27:14.493376 10005 consensus_queue.cc:260] T 00000000000000000000000000000000 P 07cb12b3ca0b492bbfa9c463582d2e3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07cb12b3ca0b492bbfa9c463582d2e3a" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 40703 } }
I20260810 03:27:14.493599 10005 raft_consensus.cc:388] T 00000000000000000000000000000000 P 07cb12b3ca0b492bbfa9c463582d2e3a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 03:27:14.493734 10005 raft_consensus.cc:482] T 00000000000000000000000000000000 P 07cb12b3ca0b492bbfa9c463582d2e3a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 03:27:14.493896 10005 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 07cb12b3ca0b492bbfa9c463582d2e3a [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:27:14.496320 10005 raft_consensus.cc:504] T 00000000000000000000000000000000 P 07cb12b3ca0b492bbfa9c463582d2e3a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07cb12b3ca0b492bbfa9c463582d2e3a" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 40703 } }
I20260810 03:27:14.496910 10005 leader_election.cc:302] T 00000000000000000000000000000000 P 07cb12b3ca0b492bbfa9c463582d2e3a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 07cb12b3ca0b492bbfa9c463582d2e3a; no voters:
I20260810 03:27:14.497536 10005 leader_election.cc:288] T 00000000000000000000000000000000 P 07cb12b3ca0b492bbfa9c463582d2e3a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 03:27:14.497685 10012 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 07cb12b3ca0b492bbfa9c463582d2e3a [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:27:14.498451 10012 raft_consensus.cc:686] T 00000000000000000000000000000000 P 07cb12b3ca0b492bbfa9c463582d2e3a [term 1 LEADER]: Becoming Leader. State: Replica: 07cb12b3ca0b492bbfa9c463582d2e3a, State: Running, Role: LEADER
I20260810 03:27:14.498983 10012 consensus_queue.cc:237] T 00000000000000000000000000000000 P 07cb12b3ca0b492bbfa9c463582d2e3a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07cb12b3ca0b492bbfa9c463582d2e3a" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 40703 } }
I20260810 03:27:14.499733 10005 sys_catalog.cc:564] T 00000000000000000000000000000000 P 07cb12b3ca0b492bbfa9c463582d2e3a [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:27:14.501655 10016 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07cb12b3ca0b492bbfa9c463582d2e3a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 07cb12b3ca0b492bbfa9c463582d2e3a. Latest consensus state: current_term: 1 leader_uuid: "07cb12b3ca0b492bbfa9c463582d2e3a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07cb12b3ca0b492bbfa9c463582d2e3a" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 40703 } } }
I20260810 03:27:14.502218 10016 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07cb12b3ca0b492bbfa9c463582d2e3a [sys.catalog]: This master's current role is: LEADER
I20260810 03:27:14.502269 10013 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07cb12b3ca0b492bbfa9c463582d2e3a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "07cb12b3ca0b492bbfa9c463582d2e3a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07cb12b3ca0b492bbfa9c463582d2e3a" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 40703 } } }
I20260810 03:27:14.502717 10013 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07cb12b3ca0b492bbfa9c463582d2e3a [sys.catalog]: This master's current role is: LEADER
I20260810 03:27:14.505208 10021 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 03:27:14.511559 10021 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 03:27:14.521168 10021 catalog_manager.cc:1347] Generated new cluster ID: 7357000c666c4a5b9ee764faa7254444
I20260810 03:27:14.521307 10021 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 03:27:14.535485 10021 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 03:27:14.536861 10021 catalog_manager.cc:1504] Loading token signing keys...
I20260810 03:27:14.550791 10021 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 07cb12b3ca0b492bbfa9c463582d2e3a: Generated new TSK 0
I20260810 03:27:14.551820 10021 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 03:27:14.556994 28292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iBH5/build/asan/bin/kudu
/tmp/dist-test-task24iBH5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786332336319544-28292-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786332336319544-28292-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786332336319544-28292-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786332336319544-28292-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.161.1:0
--local_ip_for_outbound_sockets=127.27.161.1
--webserver_interface=127.27.161.1
--webserver_port=0
--tserver_master_addrs=127.27.161.62:40703
--builtin_ntp_servers=127.27.161.20:39911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260810 03:27:14.925274 10040 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:27:14.925597 10040 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:27:14.925725 10040 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:27:14.933316 10040 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:27:14.933439 10040 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:27:14.933595 10040 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.161.1
I20260810 03:27:14.943156 10040 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.161.20:39911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786332336319544-28292-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786332336319544-28292-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.161.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786332336319544-28292-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.161.1
--webserver_port=0
--tserver_master_addrs=127.27.161.62:40703
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.161.1
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786332336319544-28292-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:27:14.944576 10040 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:27:14.946969 10040 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:27:14.960777 10048 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:27:14.960840 10049 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:27:14.960834 10052 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:27:14.961678 10040 server_base.cc:1034] running on GCE node
I20260810 03:27:14.962368 10040 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:27:14.963439 10040 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:27:14.964686 10040 hybrid_clock.cc:648] HybridClock initialized: now 1786332434964617 us; error 38 us; skew 500 ppm
I20260810 03:27:14.965052 10040 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 03:27:14.967924 10040 webserver.cc:460] Webserver started at http://127.27.161.1:36795/ using document root <none> and password file <none>
I20260810 03:27:14.968771 10040 fs_manager.cc:359] Metadata directory not provided
I20260810 03:27:14.968889 10040 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:27:14.969239 10040 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:27:14.972716 10040 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786332336319544-28292-0/minicluster-data/ts-0/data/instance:
uuid: "928d26251a724ed8bd9a0c87aee387c7"
format_stamp: "Formatted at 2026-08-10 03:27:14 on dist-test-slave-vpvm"
server_key: "b2e2be641a6a97c43babcb91acc1e835"
server_key_iv: "6080449ac2385a7d7d7852f64a24590f"
server_key_version: "encryptionkey@0"
I20260810 03:27:14.973672 10040 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786332336319544-28292-0/minicluster-data/ts-0/wal/instance:
uuid: "928d26251a724ed8bd9a0c87aee387c7"
format_stamp: "Formatted at 2026-08-10 03:27:14 on dist-test-slave-vpvm"
server_key: "b2e2be641a6a97c43babcb91acc1e835"
server_key_iv: "6080449ac2385a7d7d7852f64a24590f"
server_key_version: "encryptionkey@0"
I20260810 03:27:14.979988 10040 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260810 03:27:14.984272 10059 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:27:14.986131 10040 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260810 03:27:14.986507 10040 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786332336319544-28292-0/minicluster-data/ts-0/data,/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786332336319544-28292-0/minicluster-data/ts-0/wal
uuid: "928d26251a724ed8bd9a0c87aee387c7"
format_stamp: "Formatted at 2026-08-10 03:27:14 on dist-test-slave-vpvm"
server_key: "b2e2be641a6a97c43babcb91acc1e835"
server_key_iv: "6080449ac2385a7d7d7852f64a24590f"
server_key_version: "encryptionkey@0"
I20260810 03:27:14.987097 10040 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786332336319544-28292-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786332336319544-28292-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786332336319544-28292-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:27:15.014497 10040 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:27:15.015462 10040 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:27:15.015792 10040 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:27:15.016886 10040 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:27:15.018595 10040 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:27:15.018715 10040 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:27:15.018826 10040 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:27:15.018896 10040 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:27:15.053696 10040 rpc_server.cc:307] RPC server started. Bound to: 127.27.161.1:33819
I20260810 03:27:15.053766 10228 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.161.1:33819 every 8 connection(s)
I20260810 03:27:15.055081 10040 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786332336319544-28292-0/minicluster-data/ts-0/data/info.pb
I20260810 03:27:15.060204 28292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iBH5/build/asan/bin/kudu as pid 10040
I20260810 03:27:15.060407 28292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786332336319544-28292-0/minicluster-data/ts-0/wal/instance
I20260810 03:27:15.061043 28292 external_mini_cluster.cc:1428] Setting key 98c8944e3040bdee1181e1bb86ebc21f
I20260810 03:27:15.064450 28292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iBH5/build/asan/bin/kudu
/tmp/dist-test-task24iBH5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786332336319544-28292-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786332336319544-28292-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786332336319544-28292-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786332336319544-28292-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.161.2:0
--local_ip_for_outbound_sockets=127.27.161.2
--webserver_interface=127.27.161.2
--webserver_port=0
--tserver_master_addrs=127.27.161.62:40703
--builtin_ntp_servers=127.27.161.20:39911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260810 03:27:15.069868 10231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:27:15.057888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.161.1:52243 (local address 127.27.161.62:40703)
0810 03:27:15.058398 (+ 510us) server_negotiation.cc:206] Beginning negotiation
0810 03:27:15.058403 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:27:15.059688 (+ 1285us) server_negotiation.cc:407] Connection header received
0810 03:27:15.061684 (+ 1996us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:27:15.061696 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:27:15.061820 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:27:15.061973 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:27:15.063511 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:27:15.064626 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:27:15.066538 (+ 1912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:27:15.067120 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:27:15.067271 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:27:15.068835 (+ 1564us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:27:15.068842 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:27:15.068846 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:27:15.068874 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:27:15.068926 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:27:15.068942 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:27:15.069041 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0810 03:27:15.069297 (+ 256us) server_negotiation.cc:299] Negotiation successful
0810 03:27:15.069406 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":390,"thread_start_us":244,"threads_started":1}
I20260810 03:27:15.070819 10232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:27:15.058540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:40703 (local address 127.27.161.1:52243)
0810 03:27:15.059514 (+ 974us) negotiation.cc:107] Waiting for socket to connect
0810 03:27:15.059551 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0810 03:27:15.061368 (+ 1817us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:27:15.062255 (+ 887us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:27:15.062267 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:27:15.062401 (+ 134us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:27:15.063302 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:27:15.063327 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:27:15.064863 (+ 1536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:27:15.064868 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:27:15.066319 (+ 1451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:27:15.066336 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:27:15.067413 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:27:15.067421 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:27:15.067666 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:27:15.068413 (+ 747us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 03:27:15.068416 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:27:15.068474 (+ 58us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 03:27:15.068581 (+ 107us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:27:15.069059 (+ 478us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:27:15.069070 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:27:15.069077 (+ 7us) client_negotiation.cc:769] Sending connection context
0810 03:27:15.069241 (+ 164us) client_negotiation.cc:240] Negotiation successful
0810 03:27:15.069612 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":276,"threads_started":1}
I20260810 03:27:15.072566 10229 heartbeater.cc:347] Connected to a master server at 127.27.161.62:40703
I20260810 03:27:15.072892 10229 heartbeater.cc:464] Registering TS with master...
I20260810 03:27:15.073737 10229 heartbeater.cc:511] Master 127.27.161.62:40703 requested a full tablet report, sending...
I20260810 03:27:15.076135 9960 ts_manager.cc:194] Registered new tserver with Master: 928d26251a724ed8bd9a0c87aee387c7 (127.27.161.1:33819)
I20260810 03:27:15.077739 9960 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.161.1:52243
W20260810 03:27:15.421435 10233 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:27:15.421717 10233 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:27:15.421829 10233 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:27:15.429360 10233 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:27:15.429473 10233 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:27:15.429607 10233 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.161.2
I20260810 03:27:15.439190 10233 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.161.20:39911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786332336319544-28292-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786332336319544-28292-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.161.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786332336319544-28292-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.161.2
--webserver_port=0
--tserver_master_addrs=127.27.161.62:40703
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.161.2
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786332336319544-28292-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:27:15.440488 10233 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:27:15.442301 10233 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:27:15.455171 10241 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:27:15.455212 10240 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:27:15.455111 10245 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:27:15.456271 10233 server_base.cc:1034] running on GCE node
I20260810 03:27:15.456979 10233 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:27:15.458184 10233 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:27:15.459455 10233 hybrid_clock.cc:648] HybridClock initialized: now 1786332435459366 us; error 73 us; skew 500 ppm
I20260810 03:27:15.459803 10233 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 03:27:15.462416 10233 webserver.cc:460] Webserver started at http://127.27.161.2:42193/ using document root <none> and password file <none>
I20260810 03:27:15.463277 10233 fs_manager.cc:359] Metadata directory not provided
I20260810 03:27:15.463385 10233 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:27:15.463728 10233 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:27:15.467597 10233 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786332336319544-28292-0/minicluster-data/ts-1/data/instance:
uuid: "73bd977ef32c439f93adb824ef7ee6fa"
format_stamp: "Formatted at 2026-08-10 03:27:15 on dist-test-slave-vpvm"
server_key: "7d67e5b3dbfafdee4a75014136f78376"
server_key_iv: "0dcc09a5fc76ea1eead42d4268d00bc7"
server_key_version: "encryptionkey@0"
I20260810 03:27:15.468681 10233 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786332336319544-28292-0/minicluster-data/ts-1/wal/instance:
uuid: "73bd977ef32c439f93adb824ef7ee6fa"
format_stamp: "Formatted at 2026-08-10 03:27:15 on dist-test-slave-vpvm"
server_key: "7d67e5b3dbfafdee4a75014136f78376"
server_key_iv: "0dcc09a5fc76ea1eead42d4268d00bc7"
server_key_version: "encryptionkey@0"
I20260810 03:27:15.475268 10233 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260810 03:27:15.478960 10253 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:27:15.480228 10233 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260810 03:27:15.480388 10233 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786332336319544-28292-0/minicluster-data/ts-1/data,/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786332336319544-28292-0/minicluster-data/ts-1/wal
uuid: "73bd977ef32c439f93adb824ef7ee6fa"
format_stamp: "Formatted at 2026-08-10 03:27:15 on dist-test-slave-vpvm"
server_key: "7d67e5b3dbfafdee4a75014136f78376"
server_key_iv: "0dcc09a5fc76ea1eead42d4268d00bc7"
server_key_version: "encryptionkey@0"
I20260810 03:27:15.480584 10233 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786332336319544-28292-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786332336319544-28292-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786332336319544-28292-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:27:15.503985 10233 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:27:15.504958 10233 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:27:15.505439 10233 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:27:15.506734 10233 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:27:15.508491 10233 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:27:15.508600 10233 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:27:15.508723 10233 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:27:15.508795 10233 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:27:15.543680 10233 rpc_server.cc:307] RPC server started. Bound to: 127.27.161.2:35933
I20260810 03:27:15.543764 10423 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.161.2:35933 every 8 connection(s)
I20260810 03:27:15.545033 10233 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786332336319544-28292-0/minicluster-data/ts-1/data/info.pb
I20260810 03:27:15.548734 28292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iBH5/build/asan/bin/kudu as pid 10233
I20260810 03:27:15.548938 28292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786332336319544-28292-0/minicluster-data/ts-1/wal/instance
I20260810 03:27:15.549499 28292 external_mini_cluster.cc:1428] Setting key 574dcf99f1d0d7c4605f2b6b1cdda95c
I20260810 03:27:15.552482 28292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iBH5/build/asan/bin/kudu
/tmp/dist-test-task24iBH5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786332336319544-28292-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786332336319544-28292-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786332336319544-28292-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786332336319544-28292-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.161.3:0
--local_ip_for_outbound_sockets=127.27.161.3
--webserver_interface=127.27.161.3
--webserver_port=0
--tserver_master_addrs=127.27.161.62:40703
--builtin_ntp_servers=127.27.161.20:39911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260810 03:27:15.558266 10231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:27:15.547691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.161.2:48741 (local address 127.27.161.62:40703)
0810 03:27:15.547879 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0810 03:27:15.547883 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:27:15.549483 (+ 1600us) server_negotiation.cc:407] Connection header received
0810 03:27:15.551627 (+ 2144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:27:15.551633 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:27:15.551716 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:27:15.551804 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:27:15.552981 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:27:15.553707 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:27:15.555319 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:27:15.555782 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:27:15.555913 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:27:15.557506 (+ 1593us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:27:15.557513 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:27:15.557516 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:27:15.557535 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:27:15.557568 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:27:15.557581 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:27:15.557666 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0810 03:27:15.557900 (+ 234us) server_negotiation.cc:299] Negotiation successful
0810 03:27:15.558019 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260810 03:27:15.559602 10431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:27:15.548095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:40703 (local address 127.27.161.2:48741)
0810 03:27:15.549266 (+ 1171us) negotiation.cc:107] Waiting for socket to connect
0810 03:27:15.549327 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0810 03:27:15.551314 (+ 1987us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:27:15.552050 (+ 736us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:27:15.552059 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:27:15.552157 (+ 98us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:27:15.552818 (+ 661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:27:15.552838 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:27:15.553868 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:27:15.553871 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:27:15.555176 (+ 1305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:27:15.555189 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:27:15.556040 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:27:15.556048 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:27:15.556333 (+ 285us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:27:15.557098 (+ 765us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 03:27:15.557102 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:27:15.557156 (+ 54us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 03:27:15.557269 (+ 113us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:27:15.557682 (+ 413us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:27:15.557692 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:27:15.557700 (+ 8us) client_negotiation.cc:769] Sending connection context
0810 03:27:15.557879 (+ 179us) client_negotiation.cc:240] Negotiation successful
0810 03:27:15.558305 (+ 426us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":559,"thread_start_us":332,"threads_started":1}
I20260810 03:27:15.561069 10424 heartbeater.cc:347] Connected to a master server at 127.27.161.62:40703
I20260810 03:27:15.561415 10424 heartbeater.cc:464] Registering TS with master...
I20260810 03:27:15.562361 10424 heartbeater.cc:511] Master 127.27.161.62:40703 requested a full tablet report, sending...
I20260810 03:27:15.564023 9960 ts_manager.cc:194] Registered new tserver with Master: 73bd977ef32c439f93adb824ef7ee6fa (127.27.161.2:35933)
I20260810 03:27:15.564874 9960 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.161.2:48741
W20260810 03:27:15.910692 10432 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:27:15.911023 10432 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:27:15.911126 10432 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:27:15.918756 10432 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:27:15.918884 10432 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:27:15.918999 10432 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.161.3
I20260810 03:27:15.930301 10432 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.161.20:39911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786332336319544-28292-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786332336319544-28292-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.161.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786332336319544-28292-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.161.3
--webserver_port=0
--tserver_master_addrs=127.27.161.62:40703
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.161.3
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786332336319544-28292-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:27:15.931787 10432 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:27:15.933812 10432 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:27:15.948129 10439 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:27:15.948094 10444 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:27:15.948053 10438 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:27:15.948918 10432 server_base.cc:1034] running on GCE node
I20260810 03:27:15.949666 10432 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:27:15.950899 10432 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:27:15.952134 10432 hybrid_clock.cc:648] HybridClock initialized: now 1786332435952075 us; error 59 us; skew 500 ppm
I20260810 03:27:15.952502 10432 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 03:27:15.955061 10432 webserver.cc:460] Webserver started at http://127.27.161.3:38393/ using document root <none> and password file <none>
I20260810 03:27:15.955847 10432 fs_manager.cc:359] Metadata directory not provided
I20260810 03:27:15.955968 10432 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:27:15.956319 10432 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:27:15.959600 10432 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786332336319544-28292-0/minicluster-data/ts-2/data/instance:
uuid: "2625cb511c4f4a93b41441b36acf996a"
format_stamp: "Formatted at 2026-08-10 03:27:15 on dist-test-slave-vpvm"
server_key: "5848ddb1fb334cb2cf0ba7ce7cf2fcf2"
server_key_iv: "b9e6a987de236458403a4ea2cbf46480"
server_key_version: "encryptionkey@0"
I20260810 03:27:15.960518 10432 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786332336319544-28292-0/minicluster-data/ts-2/wal/instance:
uuid: "2625cb511c4f4a93b41441b36acf996a"
format_stamp: "Formatted at 2026-08-10 03:27:15 on dist-test-slave-vpvm"
server_key: "5848ddb1fb334cb2cf0ba7ce7cf2fcf2"
server_key_iv: "b9e6a987de236458403a4ea2cbf46480"
server_key_version: "encryptionkey@0"
I20260810 03:27:15.966934 10432 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.006s
I20260810 03:27:15.970820 10451 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:27:15.972266 10432 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 03:27:15.972455 10432 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786332336319544-28292-0/minicluster-data/ts-2/data,/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786332336319544-28292-0/minicluster-data/ts-2/wal
uuid: "2625cb511c4f4a93b41441b36acf996a"
format_stamp: "Formatted at 2026-08-10 03:27:15 on dist-test-slave-vpvm"
server_key: "5848ddb1fb334cb2cf0ba7ce7cf2fcf2"
server_key_iv: "b9e6a987de236458403a4ea2cbf46480"
server_key_version: "encryptionkey@0"
I20260810 03:27:15.972684 10432 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786332336319544-28292-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786332336319544-28292-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786332336319544-28292-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:27:15.987915 10432 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:27:15.988901 10432 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:27:15.989235 10432 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:27:15.990309 10432 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:27:15.991921 10432 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:27:15.992024 10432 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:27:15.992142 10432 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:27:15.992230 10432 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:27:16.024832 10432 rpc_server.cc:307] RPC server started. Bound to: 127.27.161.3:34925
I20260810 03:27:16.024892 10625 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.161.3:34925 every 8 connection(s)
I20260810 03:27:16.026335 10432 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786332336319544-28292-0/minicluster-data/ts-2/data/info.pb
I20260810 03:27:16.035137 28292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iBH5/build/asan/bin/kudu as pid 10432
I20260810 03:27:16.035336 28292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786332336319544-28292-0/minicluster-data/ts-2/wal/instance
I20260810 03:27:16.035763 28292 external_mini_cluster.cc:1428] Setting key 7262f79bd1196698e5218de456d8d6d8
I20260810 03:27:16.039836 10231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:27:16.029261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.161.3:36201 (local address 127.27.161.62:40703)
0810 03:27:16.029441 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0810 03:27:16.029445 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:27:16.030959 (+ 1514us) server_negotiation.cc:407] Connection header received
0810 03:27:16.032810 (+ 1851us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:27:16.032817 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:27:16.032906 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:27:16.032995 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:27:16.034194 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:27:16.034995 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:27:16.036767 (+ 1772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:27:16.037197 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:27:16.037314 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:27:16.038982 (+ 1668us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:27:16.038995 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:27:16.038998 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:27:16.039017 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:27:16.039051 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:27:16.039064 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:27:16.039142 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0810 03:27:16.039395 (+ 253us) server_negotiation.cc:299] Negotiation successful
0810 03:27:16.039503 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260810 03:27:16.040940 10628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:27:16.029742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:40703 (local address 127.27.161.3:36201)
0810 03:27:16.030797 (+ 1055us) negotiation.cc:107] Waiting for socket to connect
0810 03:27:16.030836 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0810 03:27:16.032507 (+ 1671us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:27:16.033263 (+ 756us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:27:16.033273 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:27:16.033369 (+ 96us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:27:16.034017 (+ 648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:27:16.034038 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:27:16.035164 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:27:16.035168 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:27:16.036602 (+ 1434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:27:16.036617 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:27:16.037439 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:27:16.037447 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:27:16.037699 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:27:16.038565 (+ 866us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 03:27:16.038569 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:27:16.038653 (+ 84us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 03:27:16.038779 (+ 126us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:27:16.039161 (+ 382us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:27:16.039173 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:27:16.039182 (+ 9us) client_negotiation.cc:769] Sending connection context
0810 03:27:16.039351 (+ 169us) client_negotiation.cc:240] Negotiation successful
0810 03:27:16.039803 (+ 452us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":563,"thread_start_us":356,"threads_started":1}
I20260810 03:27:16.042373 10626 heartbeater.cc:347] Connected to a master server at 127.27.161.62:40703
I20260810 03:27:16.042658 10626 heartbeater.cc:464] Registering TS with master...
I20260810 03:27:16.043457 10626 heartbeater.cc:511] Master 127.27.161.62:40703 requested a full tablet report, sending...
I20260810 03:27:16.044881 9960 ts_manager.cc:194] Registered new tserver with Master: 2625cb511c4f4a93b41441b36acf996a (127.27.161.3:34925)
I20260810 03:27:16.045696 9960 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.161.3:36201
I20260810 03:27:16.051267 28292 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260810 03:27:16.057898 10231 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:27:16.056191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50824 (local address 127.27.161.62:40703)
0810 03:27:16.056419 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0810 03:27:16.056427 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 03:27:16.056920 (+ 493us) server_negotiation.cc:407] Connection header received
0810 03:27:16.057082 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:27:16.057086 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:27:16.057164 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:27:16.057246 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:27:16.057647 (+ 401us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50824: BlockingRecv error: recv got EOF from 127.0.0.1:50824 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
I20260810 03:27:16.058790 28292 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iBH5/build/asan/bin/kudu pid 10040
I20260810 03:27:16.067601 10639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:27:16.059865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50574 (local address 127.27.161.1:33819)
0810 03:27:16.060401 (+ 536us) server_negotiation.cc:206] Beginning negotiation
0810 03:27:16.060411 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 03:27:16.060445 (+ 34us) server_negotiation.cc:407] Connection header received
0810 03:27:16.060515 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:27:16.060522 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:27:16.060794 (+ 272us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:27:16.060940 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:27:16.061373 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:27:16.062487 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:27:16.064044 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:27:16.064856 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:27:16.064971 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:27:16.066134 (+ 1163us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:27:16.066144 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:27:16.066161 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:27:16.066783 (+ 622us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:27:16.066838 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:27:16.066860 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:27:16.066955 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0810 03:27:16.067133 (+ 178us) server_negotiation.cc:299] Negotiation successful
0810 03:27:16.067266 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":382,"thread_start_us":256,"threads_started":1}
I20260810 03:27:16.068924 10223 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:27:16.152890 10229 heartbeater.cc:503] Master 127.27.161.62:40703 was elected leader, sending a full tablet report...
I20260810 03:27:16.154235 28292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iBH5/build/asan/bin/kudu with pid 10040
I20260810 03:27:16.179226 28292 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iBH5/build/asan/bin/kudu pid 10233
W20260810 03:27:16.179045 9939 connection.cc:535] server connection from 127.27.161.1:52243 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260810 03:27:16.187434 10645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:27:16.180145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54012 (local address 127.27.161.2:35933)
0810 03:27:16.180652 (+ 507us) server_negotiation.cc:206] Beginning negotiation
0810 03:27:16.180668 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 03:27:16.180695 (+ 27us) server_negotiation.cc:407] Connection header received
0810 03:27:16.180757 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:27:16.180763 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:27:16.180979 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:27:16.181126 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:27:16.181548 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:27:16.182639 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:27:16.183906 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:27:16.184480 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:27:16.184600 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:27:16.185648 (+ 1048us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:27:16.185665 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:27:16.185680 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:27:16.186513 (+ 833us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:27:16.186572 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:27:16.186595 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:27:16.186718 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0810 03:27:16.186891 (+ 173us) server_negotiation.cc:299] Negotiation successful
0810 03:27:16.187043 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":365,"thread_start_us":247,"threads_started":1}
I20260810 03:27:16.188856 10418 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:27:16.273866 28292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iBH5/build/asan/bin/kudu with pid 10233
I20260810 03:27:16.299587 28292 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iBH5/build/asan/bin/kudu pid 10432
I20260810 03:27:16.307951 10647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:27:16.300571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59316 (local address 127.27.161.3:34925)
0810 03:27:16.301303 (+ 732us) server_negotiation.cc:206] Beginning negotiation
0810 03:27:16.301310 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:27:16.301342 (+ 32us) server_negotiation.cc:407] Connection header received
0810 03:27:16.301408 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:27:16.301415 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:27:16.301631 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:27:16.301759 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:27:16.302206 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:27:16.303268 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:27:16.304470 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:27:16.305101 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:27:16.305210 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:27:16.306456 (+ 1246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:27:16.306465 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:27:16.306480 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:27:16.307075 (+ 595us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:27:16.307124 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:27:16.307163 (+ 39us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:27:16.307254 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0810 03:27:16.307418 (+ 164us) server_negotiation.cc:299] Negotiation successful
0810 03:27:16.307556 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":571,"thread_start_us":396,"threads_started":1}
I20260810 03:27:16.309257 10617 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:27:16.390633 28292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iBH5/build/asan/bin/kudu with pid 10432
I20260810 03:27:16.415484 28292 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iBH5/build/asan/bin/kudu pid 9913
I20260810 03:27:16.416776 9996 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:27:16.522541 28292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iBH5/build/asan/bin/kudu with pid 9913
2026-08-10T03:27:16Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2629 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-08-10T03:27:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T03:27:16Z Disabled control of system clock
I20260810 03:27:16.623606 28292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iBH5/build/asan/bin/kudu
/tmp/dist-test-task24iBH5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786332336319544-28292-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786332336319544-28292-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786332336319544-28292-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786332336319544-28292-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.161.62:35967
--webserver_interface=127.27.161.62
--webserver_port=0
--builtin_ntp_servers=127.27.161.20:40923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.161.62:35967
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260810 03:27:16.980181 10655 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:27:16.980522 10655 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:27:16.980664 10655 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:27:16.988210 10655 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:27:16.988363 10655 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:27:16.988433 10655 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:27:16.988516 10655 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:27:16.988579 10655 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:27:16.998219 10655 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.161.20:40923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786332336319544-28292-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786332336319544-28292-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.161.62:35967
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.161.62:35967
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786332336319544-28292-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.161.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786332336319544-28292-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:27:16.999614 10655 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:27:17.001552 10655 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:27:17.013427 10663 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:27:17.013447 10667 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:27:17.013478 10664 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:27:17.014262 10655 server_base.cc:1034] running on GCE node
I20260810 03:27:17.015095 10655 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:27:17.016624 10655 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:27:17.017839 10655 hybrid_clock.cc:648] HybridClock initialized: now 1786332437017793 us; error 39 us; skew 500 ppm
I20260810 03:27:17.018229 10655 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 03:27:17.020696 10655 webserver.cc:460] Webserver started at http://127.27.161.62:41103/ using document root <none> and password file <none>
I20260810 03:27:17.021453 10655 fs_manager.cc:359] Metadata directory not provided
I20260810 03:27:17.021575 10655 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:27:17.021916 10655 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:27:17.027539 10655 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786332336319544-28292-0/minicluster-data/master-0/data/instance:
uuid: "6d842f36273546378c0a3879e1b0f3cf"
format_stamp: "Formatted at 2026-08-10 03:27:17 on dist-test-slave-vpvm"
server_key: "2049ce96f6df158ecf1fe8858ccda13c"
server_key_iv: "9002d7e91e9348533df91443cd74b39e"
server_key_version: "encryptionkey@0"
I20260810 03:27:17.028728 10655 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786332336319544-28292-0/minicluster-data/master-0/wal/instance:
uuid: "6d842f36273546378c0a3879e1b0f3cf"
format_stamp: "Formatted at 2026-08-10 03:27:17 on dist-test-slave-vpvm"
server_key: "2049ce96f6df158ecf1fe8858ccda13c"
server_key_iv: "9002d7e91e9348533df91443cd74b39e"
server_key_version: "encryptionkey@0"
I20260810 03:27:17.034458 10655 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260810 03:27:17.038069 10677 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:27:17.039840 10655 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 03:27:17.040005 10655 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786332336319544-28292-0/minicluster-data/master-0/data,/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786332336319544-28292-0/minicluster-data/master-0/wal
uuid: "6d842f36273546378c0a3879e1b0f3cf"
format_stamp: "Formatted at 2026-08-10 03:27:17 on dist-test-slave-vpvm"
server_key: "2049ce96f6df158ecf1fe8858ccda13c"
server_key_iv: "9002d7e91e9348533df91443cd74b39e"
server_key_version: "encryptionkey@0"
I20260810 03:27:17.040213 10655 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786332336319544-28292-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786332336319544-28292-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786332336319544-28292-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:27:17.046154 10655 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:27:17.046947 10655 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:27:17.047219 10655 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:27:17.065176 10655 rpc_server.cc:307] RPC server started. Bound to: 127.27.161.62:35967
I20260810 03:27:17.065183 10759 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.161.62:35967 every 8 connection(s)
I20260810 03:27:17.066864 10655 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786332336319544-28292-0/minicluster-data/master-0/data/info.pb
I20260810 03:27:17.071264 10760 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:27:17.076092 28292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iBH5/build/asan/bin/kudu as pid 10655
I20260810 03:27:17.076318 28292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786332336319544-28292-0/minicluster-data/master-0/wal/instance
I20260810 03:27:17.077124 28292 external_mini_cluster.cc:1428] Setting key 0a63e4bcdcf53fa4e535c2afa6e78b16
I20260810 03:27:17.083940 10760 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d842f36273546378c0a3879e1b0f3cf: Bootstrap starting.
I20260810 03:27:17.086560 10765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:27:17.079473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39014 (local address 127.27.161.62:35967)
0810 03:27:17.080746 (+ 1273us) server_negotiation.cc:206] Beginning negotiation
0810 03:27:17.080759 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 03:27:17.080810 (+ 51us) server_negotiation.cc:407] Connection header received
0810 03:27:17.082705 (+ 1895us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:27:17.082735 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:27:17.083234 (+ 499us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:27:17.083638 (+ 404us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:27:17.084769 (+ 1131us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:27:17.084776 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:27:17.084790 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:27:17.084807 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:27:17.084878 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:27:17.084895 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:27:17.085125 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0810 03:27:17.085270 (+ 145us) server_negotiation.cc:299] Negotiation successful
0810 03:27:17.085569 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":662,"thread_start_us":401,"threads_started":1}
I20260810 03:27:17.087474 10760 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6d842f36273546378c0a3879e1b0f3cf: Neither blocks nor log segments found. Creating new log.
I20260810 03:27:17.088564 10760 log.cc:824] T 00000000000000000000000000000000 P 6d842f36273546378c0a3879e1b0f3cf: Log is configured to *not* fsync() on all Append() calls
I20260810 03:27:17.091451 10760 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d842f36273546378c0a3879e1b0f3cf: No bootstrap required, opened a new log
I20260810 03:27:17.098688 10760 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6d842f36273546378c0a3879e1b0f3cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d842f36273546378c0a3879e1b0f3cf" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 35967 } }
I20260810 03:27:17.099090 10760 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6d842f36273546378c0a3879e1b0f3cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:27:17.099184 10760 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6d842f36273546378c0a3879e1b0f3cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d842f36273546378c0a3879e1b0f3cf, State: Initialized, Role: FOLLOWER
I20260810 03:27:17.099789 10760 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6d842f36273546378c0a3879e1b0f3cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d842f36273546378c0a3879e1b0f3cf" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 35967 } }
I20260810 03:27:17.100024 10760 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6d842f36273546378c0a3879e1b0f3cf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 03:27:17.100158 10760 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6d842f36273546378c0a3879e1b0f3cf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 03:27:17.100323 10760 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6d842f36273546378c0a3879e1b0f3cf [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:27:17.102813 10760 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6d842f36273546378c0a3879e1b0f3cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d842f36273546378c0a3879e1b0f3cf" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 35967 } }
I20260810 03:27:17.103371 10760 leader_election.cc:302] T 00000000000000000000000000000000 P 6d842f36273546378c0a3879e1b0f3cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6d842f36273546378c0a3879e1b0f3cf; no voters:
I20260810 03:27:17.104023 10760 leader_election.cc:288] T 00000000000000000000000000000000 P 6d842f36273546378c0a3879e1b0f3cf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 03:27:17.104192 10767 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6d842f36273546378c0a3879e1b0f3cf [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:27:17.105017 10767 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6d842f36273546378c0a3879e1b0f3cf [term 1 LEADER]: Becoming Leader. State: Replica: 6d842f36273546378c0a3879e1b0f3cf, State: Running, Role: LEADER
I20260810 03:27:17.105679 10767 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6d842f36273546378c0a3879e1b0f3cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d842f36273546378c0a3879e1b0f3cf" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 35967 } }
I20260810 03:27:17.106349 10760 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6d842f36273546378c0a3879e1b0f3cf [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:27:17.108538 10769 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d842f36273546378c0a3879e1b0f3cf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6d842f36273546378c0a3879e1b0f3cf. Latest consensus state: current_term: 1 leader_uuid: "6d842f36273546378c0a3879e1b0f3cf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d842f36273546378c0a3879e1b0f3cf" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 35967 } } }
I20260810 03:27:17.108939 10769 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d842f36273546378c0a3879e1b0f3cf [sys.catalog]: This master's current role is: LEADER
I20260810 03:27:17.109350 10768 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d842f36273546378c0a3879e1b0f3cf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6d842f36273546378c0a3879e1b0f3cf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d842f36273546378c0a3879e1b0f3cf" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 35967 } } }
I20260810 03:27:17.109879 10768 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d842f36273546378c0a3879e1b0f3cf [sys.catalog]: This master's current role is: LEADER
I20260810 03:27:17.111931 10775 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 03:27:17.118006 10775 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 03:27:17.128906 10775 catalog_manager.cc:1347] Generated new cluster ID: fd2f3e6214a949908da3a22e669a9fcf
I20260810 03:27:17.129050 10775 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 03:27:17.144110 10775 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 03:27:17.144802 10775 catalog_manager.cc:1504] Loading token signing keys...
I20260810 03:27:17.152805 10775 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6d842f36273546378c0a3879e1b0f3cf: Generated new TSK 0
I20260810 03:27:17.153735 10775 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 03:27:17.162793 28292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iBH5/build/asan/bin/kudu
/tmp/dist-test-task24iBH5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786332336319544-28292-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786332336319544-28292-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786332336319544-28292-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786332336319544-28292-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.161.1:0
--local_ip_for_outbound_sockets=127.27.161.1
--webserver_interface=127.27.161.1
--webserver_port=0
--tserver_master_addrs=127.27.161.62:35967
--builtin_ntp_servers=127.27.161.20:40923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260810 03:27:17.555622 10793 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:27:17.555920 10793 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:27:17.556013 10793 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:27:17.563917 10793 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:27:17.564034 10793 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:27:17.564157 10793 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.161.1
I20260810 03:27:17.573450 10793 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.161.20:40923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786332336319544-28292-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786332336319544-28292-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.161.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786332336319544-28292-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.161.1
--webserver_port=0
--tserver_master_addrs=127.27.161.62:35967
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.161.1
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786332336319544-28292-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:27:17.574796 10793 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:27:17.576599 10793 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:27:17.590543 10800 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:27:17.590610 10803 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:27:17.590575 10799 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:27:17.591317 10793 server_base.cc:1034] running on GCE node
I20260810 03:27:17.591997 10793 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:27:17.593075 10793 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:27:17.594296 10793 hybrid_clock.cc:648] HybridClock initialized: now 1786332437594233 us; error 55 us; skew 500 ppm
I20260810 03:27:17.594650 10793 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 03:27:17.597106 10793 webserver.cc:460] Webserver started at http://127.27.161.1:35577/ using document root <none> and password file <none>
I20260810 03:27:17.597869 10793 fs_manager.cc:359] Metadata directory not provided
I20260810 03:27:17.598018 10793 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:27:17.598349 10793 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:27:17.601387 10793 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786332336319544-28292-0/minicluster-data/ts-0/data/instance:
uuid: "cf814739976d4135a23b99880370f8ca"
format_stamp: "Formatted at 2026-08-10 03:27:17 on dist-test-slave-vpvm"
server_key: "492821f7910a42c75af3bdc1dccee218"
server_key_iv: "eab3f47980231c8be4d5804720154f58"
server_key_version: "encryptionkey@0"
I20260810 03:27:17.602281 10793 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786332336319544-28292-0/minicluster-data/ts-0/wal/instance:
uuid: "cf814739976d4135a23b99880370f8ca"
format_stamp: "Formatted at 2026-08-10 03:27:17 on dist-test-slave-vpvm"
server_key: "492821f7910a42c75af3bdc1dccee218"
server_key_iv: "eab3f47980231c8be4d5804720154f58"
server_key_version: "encryptionkey@0"
I20260810 03:27:17.607870 10793 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260810 03:27:17.611425 10813 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:27:17.612885 10793 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260810 03:27:17.613067 10793 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786332336319544-28292-0/minicluster-data/ts-0/data,/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786332336319544-28292-0/minicluster-data/ts-0/wal
uuid: "cf814739976d4135a23b99880370f8ca"
format_stamp: "Formatted at 2026-08-10 03:27:17 on dist-test-slave-vpvm"
server_key: "492821f7910a42c75af3bdc1dccee218"
server_key_iv: "eab3f47980231c8be4d5804720154f58"
server_key_version: "encryptionkey@0"
I20260810 03:27:17.613303 10793 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786332336319544-28292-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786332336319544-28292-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786332336319544-28292-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:27:17.619223 10793 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:27:17.620072 10793 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:27:17.620325 10793 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:27:17.621250 10793 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:27:17.622786 10793 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:27:17.622884 10793 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:27:17.622990 10793 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:27:17.623059 10793 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:27:17.654871 10793 rpc_server.cc:307] RPC server started. Bound to: 127.27.161.1:42495
I20260810 03:27:17.654927 10984 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.161.1:42495 every 8 connection(s)
I20260810 03:27:17.656213 10793 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786332336319544-28292-0/minicluster-data/ts-0/data/info.pb
I20260810 03:27:17.661250 28292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iBH5/build/asan/bin/kudu as pid 10793
I20260810 03:27:17.661427 28292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786332336319544-28292-0/minicluster-data/ts-0/wal/instance
I20260810 03:27:17.662077 28292 external_mini_cluster.cc:1428] Setting key 63020bddbb2068ed70d997ebf6e4c832
I20260810 03:27:17.664661 10988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:27:17.658746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.161.1:34507 (local address 127.27.161.62:35967)
0810 03:27:17.659164 (+ 418us) server_negotiation.cc:206] Beginning negotiation
0810 03:27:17.659169 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:27:17.660460 (+ 1291us) server_negotiation.cc:407] Connection header received
0810 03:27:17.662269 (+ 1809us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:27:17.662275 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:27:17.662346 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:27:17.662466 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:27:17.663762 (+ 1296us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:27:17.663767 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:27:17.663770 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:27:17.663786 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:27:17.663829 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:27:17.663840 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:27:17.663939 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0810 03:27:17.664292 (+ 353us) server_negotiation.cc:299] Negotiation successful
0810 03:27:17.664364 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":206,"threads_started":1}
I20260810 03:27:17.665247 28292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iBH5/build/asan/bin/kudu
/tmp/dist-test-task24iBH5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786332336319544-28292-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786332336319544-28292-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786332336319544-28292-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786332336319544-28292-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.161.2:0
--local_ip_for_outbound_sockets=127.27.161.2
--webserver_interface=127.27.161.2
--webserver_port=0
--tserver_master_addrs=127.27.161.62:35967
--builtin_ntp_servers=127.27.161.20:40923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260810 03:27:17.665855 10989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:27:17.659320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:35967 (local address 127.27.161.1:34507)
0810 03:27:17.660302 (+ 982us) negotiation.cc:107] Waiting for socket to connect
0810 03:27:17.660344 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0810 03:27:17.661972 (+ 1628us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:27:17.662653 (+ 681us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:27:17.662661 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:27:17.662748 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:27:17.663417 (+ 669us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 03:27:17.663420 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:27:17.663469 (+ 49us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 03:27:17.663577 (+ 108us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:27:17.663923 (+ 346us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:27:17.663935 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:27:17.663942 (+ 7us) client_negotiation.cc:769] Sending connection context
0810 03:27:17.664348 (+ 406us) client_negotiation.cc:240] Negotiation successful
0810 03:27:17.664627 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":259,"threads_started":1}
I20260810 03:27:17.667436 10985 heartbeater.cc:347] Connected to a master server at 127.27.161.62:35967
I20260810 03:27:17.667761 10985 heartbeater.cc:464] Registering TS with master...
I20260810 03:27:17.668375 10985 heartbeater.cc:511] Master 127.27.161.62:35967 requested a full tablet report, sending...
I20260810 03:27:17.670681 10705 ts_manager.cc:194] Registered new tserver with Master: cf814739976d4135a23b99880370f8ca (127.27.161.1:42495)
W20260810 03:27:18.022259 10992 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:27:18.022603 10992 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:27:18.022742 10992 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:27:18.030740 10992 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:27:18.030901 10992 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:27:18.031054 10992 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.161.2
I20260810 03:27:18.041280 10992 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.161.20:40923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786332336319544-28292-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786332336319544-28292-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.161.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786332336319544-28292-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.161.2
--webserver_port=0
--tserver_master_addrs=127.27.161.62:35967
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.161.2
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786332336319544-28292-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:27:18.043012 10992 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:27:18.045332 10992 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:27:18.059803 11001 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:27:18.059834 10999 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:27:18.059764 11003 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:27:18.060562 10992 server_base.cc:1034] running on GCE node
I20260810 03:27:18.061291 10992 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:27:18.062417 10992 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:27:18.063643 10992 hybrid_clock.cc:648] HybridClock initialized: now 1786332438063580 us; error 48 us; skew 500 ppm
I20260810 03:27:18.063993 10992 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 03:27:18.074276 10992 webserver.cc:460] Webserver started at http://127.27.161.2:36925/ using document root <none> and password file <none>
I20260810 03:27:18.075273 10992 fs_manager.cc:359] Metadata directory not provided
I20260810 03:27:18.075408 10992 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:27:18.075807 10992 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:27:18.080102 10992 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786332336319544-28292-0/minicluster-data/ts-1/data/instance:
uuid: "534f151b373f44b18a9cf4da653159d4"
format_stamp: "Formatted at 2026-08-10 03:27:18 on dist-test-slave-vpvm"
server_key: "6aa829315d73ded9104346b642ce0225"
server_key_iv: "93f2ecfcb322b44082271aa84a046a7b"
server_key_version: "encryptionkey@0"
I20260810 03:27:18.081243 10992 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786332336319544-28292-0/minicluster-data/ts-1/wal/instance:
uuid: "534f151b373f44b18a9cf4da653159d4"
format_stamp: "Formatted at 2026-08-10 03:27:18 on dist-test-slave-vpvm"
server_key: "6aa829315d73ded9104346b642ce0225"
server_key_iv: "93f2ecfcb322b44082271aa84a046a7b"
server_key_version: "encryptionkey@0"
I20260810 03:27:18.087327 10992 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.003s
I20260810 03:27:18.091082 11011 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:27:18.092522 10992 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 03:27:18.092696 10992 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786332336319544-28292-0/minicluster-data/ts-1/data,/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786332336319544-28292-0/minicluster-data/ts-1/wal
uuid: "534f151b373f44b18a9cf4da653159d4"
format_stamp: "Formatted at 2026-08-10 03:27:18 on dist-test-slave-vpvm"
server_key: "6aa829315d73ded9104346b642ce0225"
server_key_iv: "93f2ecfcb322b44082271aa84a046a7b"
server_key_version: "encryptionkey@0"
I20260810 03:27:18.092922 10992 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786332336319544-28292-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786332336319544-28292-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786332336319544-28292-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:27:18.098886 10992 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:27:18.099722 10992 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:27:18.099983 10992 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:27:18.100946 10992 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:27:18.102519 10992 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:27:18.102622 10992 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:27:18.102733 10992 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:27:18.102802 10992 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:27:18.135771 10992 rpc_server.cc:307] RPC server started. Bound to: 127.27.161.2:42919
I20260810 03:27:18.135846 11186 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.161.2:42919 every 8 connection(s)
I20260810 03:27:18.137130 10992 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786332336319544-28292-0/minicluster-data/ts-1/data/info.pb
I20260810 03:27:18.145764 10988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:27:18.139923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.161.2:34391 (local address 127.27.161.62:35967)
0810 03:27:18.140143 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0810 03:27:18.140147 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:27:18.141538 (+ 1391us) server_negotiation.cc:407] Connection header received
0810 03:27:18.143476 (+ 1938us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:27:18.143481 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:27:18.143559 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:27:18.143655 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:27:18.145072 (+ 1417us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:27:18.145076 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:27:18.145079 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:27:18.145096 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:27:18.145132 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:27:18.145146 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:27:18.145269 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0810 03:27:18.145472 (+ 203us) server_negotiation.cc:299] Negotiation successful
0810 03:27:18.145535 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260810 03:27:18.146786 11189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:27:18.140365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:35967 (local address 127.27.161.2:34391)
0810 03:27:18.141339 (+ 974us) negotiation.cc:107] Waiting for socket to connect
0810 03:27:18.141377 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0810 03:27:18.143116 (+ 1739us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:27:18.143920 (+ 804us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:27:18.143928 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:27:18.144043 (+ 115us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:27:18.144755 (+ 712us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 03:27:18.144758 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:27:18.144799 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 03:27:18.144897 (+ 98us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:27:18.145233 (+ 336us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:27:18.145243 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:27:18.145250 (+ 7us) client_negotiation.cc:769] Sending connection context
0810 03:27:18.145504 (+ 254us) client_negotiation.cc:240] Negotiation successful
0810 03:27:18.145746 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":504,"thread_start_us":297,"threads_started":1}
I20260810 03:27:18.147336 28292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iBH5/build/asan/bin/kudu as pid 10992
I20260810 03:27:18.147490 28292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786332336319544-28292-0/minicluster-data/ts-1/wal/instance
I20260810 03:27:18.148053 28292 external_mini_cluster.cc:1428] Setting key 4082031b7759f4f33a696c9c68e4280f
I20260810 03:27:18.148486 11187 heartbeater.cc:347] Connected to a master server at 127.27.161.62:35967
I20260810 03:27:18.148880 11187 heartbeater.cc:464] Registering TS with master...
I20260810 03:27:18.149641 11187 heartbeater.cc:511] Master 127.27.161.62:35967 requested a full tablet report, sending...
I20260810 03:27:18.151791 28292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iBH5/build/asan/bin/kudu
/tmp/dist-test-task24iBH5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786332336319544-28292-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786332336319544-28292-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786332336319544-28292-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786332336319544-28292-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.161.3:0
--local_ip_for_outbound_sockets=127.27.161.3
--webserver_interface=127.27.161.3
--webserver_port=0
--tserver_master_addrs=127.27.161.62:35967
--builtin_ntp_servers=127.27.161.20:40923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260810 03:27:18.151932 10705 ts_manager.cc:194] Registered new tserver with Master: 534f151b373f44b18a9cf4da653159d4 (127.27.161.2:42919)
W20260810 03:27:18.507732 11190 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:27:18.508061 11190 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:27:18.508159 11190 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:27:18.515753 11190 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:27:18.515877 11190 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:27:18.516003 11190 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.161.3
I20260810 03:27:18.525676 11190 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.161.20:40923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786332336319544-28292-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786332336319544-28292-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.161.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786332336319544-28292-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.161.3
--webserver_port=0
--tserver_master_addrs=127.27.161.62:35967
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.161.3
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786332336319544-28292-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:27:18.527043 11190 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:27:18.528919 11190 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:27:18.543571 11197 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:27:18.543467 11199 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:27:18.543653 11196 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:27:18.544687 11190 server_base.cc:1034] running on GCE node
I20260810 03:27:18.545308 11190 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:27:18.546392 11190 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:27:18.547616 11190 hybrid_clock.cc:648] HybridClock initialized: now 1786332438547549 us; error 61 us; skew 500 ppm
I20260810 03:27:18.547966 11190 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 03:27:18.550561 11190 webserver.cc:460] Webserver started at http://127.27.161.3:40033/ using document root <none> and password file <none>
I20260810 03:27:18.551309 11190 fs_manager.cc:359] Metadata directory not provided
I20260810 03:27:18.551406 11190 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:27:18.551714 11190 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:27:18.555320 11190 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786332336319544-28292-0/minicluster-data/ts-2/data/instance:
uuid: "123904e534fa4243b7bcb411d792ee29"
format_stamp: "Formatted at 2026-08-10 03:27:18 on dist-test-slave-vpvm"
server_key: "09673f9d864ddff7a0faaf133406eac1"
server_key_iv: "9c2080ae6664af8bb46326a9b4dc854c"
server_key_version: "encryptionkey@0"
I20260810 03:27:18.556332 11190 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786332336319544-28292-0/minicluster-data/ts-2/wal/instance:
uuid: "123904e534fa4243b7bcb411d792ee29"
format_stamp: "Formatted at 2026-08-10 03:27:18 on dist-test-slave-vpvm"
server_key: "09673f9d864ddff7a0faaf133406eac1"
server_key_iv: "9c2080ae6664af8bb46326a9b4dc854c"
server_key_version: "encryptionkey@0"
I20260810 03:27:18.562548 11190 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260810 03:27:18.566431 11208 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:27:18.567909 11190 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260810 03:27:18.568076 11190 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786332336319544-28292-0/minicluster-data/ts-2/data,/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786332336319544-28292-0/minicluster-data/ts-2/wal
uuid: "123904e534fa4243b7bcb411d792ee29"
format_stamp: "Formatted at 2026-08-10 03:27:18 on dist-test-slave-vpvm"
server_key: "09673f9d864ddff7a0faaf133406eac1"
server_key_iv: "9c2080ae6664af8bb46326a9b4dc854c"
server_key_version: "encryptionkey@0"
I20260810 03:27:18.568279 11190 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786332336319544-28292-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786332336319544-28292-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786332336319544-28292-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:27:18.574506 11190 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:27:18.575378 11190 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:27:18.575634 11190 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:27:18.576624 11190 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:27:18.578246 11190 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:27:18.578346 11190 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:27:18.578437 11190 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:27:18.578500 11190 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:27:18.613353 11190 rpc_server.cc:307] RPC server started. Bound to: 127.27.161.3:44791
I20260810 03:27:18.613409 11385 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.161.3:44791 every 8 connection(s)
I20260810 03:27:18.614949 11190 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786332336319544-28292-0/minicluster-data/ts-2/data/info.pb
I20260810 03:27:18.624732 28292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iBH5/build/asan/bin/kudu as pid 11190
I20260810 03:27:18.624929 28292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786332336319544-28292-0/minicluster-data/ts-2/wal/instance
I20260810 03:27:18.625465 28292 external_mini_cluster.cc:1428] Setting key 234d15b7ac67f5dd8ad085391e2cc0eb
I20260810 03:27:18.626300 10988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:27:18.618745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.161.3:37401 (local address 127.27.161.62:35967)
0810 03:27:18.619034 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0810 03:27:18.619042 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 03:27:18.621087 (+ 2045us) server_negotiation.cc:407] Connection header received
0810 03:27:18.623179 (+ 2092us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:27:18.623184 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:27:18.623272 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:27:18.623373 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:27:18.625390 (+ 2017us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:27:18.625397 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:27:18.625402 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:27:18.625430 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:27:18.625488 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:27:18.625507 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:27:18.625661 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0810 03:27:18.625936 (+ 275us) server_negotiation.cc:299] Negotiation successful
0810 03:27:18.626050 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104}
I20260810 03:27:18.627868 11388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:27:18.619496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:35967 (local address 127.27.161.3:37401)
0810 03:27:18.620837 (+ 1341us) negotiation.cc:107] Waiting for socket to connect
0810 03:27:18.620900 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0810 03:27:18.622806 (+ 1906us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:27:18.623773 (+ 967us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:27:18.623790 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:27:18.623933 (+ 143us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:27:18.624794 (+ 861us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 03:27:18.624797 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:27:18.624850 (+ 53us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 03:27:18.625123 (+ 273us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:27:18.625666 (+ 543us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:27:18.625678 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:27:18.625687 (+ 9us) client_negotiation.cc:769] Sending connection context
0810 03:27:18.626035 (+ 348us) client_negotiation.cc:240] Negotiation successful
0810 03:27:18.626445 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":728,"thread_start_us":483,"threads_started":1}
I20260810 03:27:18.630061 11386 heartbeater.cc:347] Connected to a master server at 127.27.161.62:35967
I20260810 03:27:18.630451 11386 heartbeater.cc:464] Registering TS with master...
I20260810 03:27:18.631230 11386 heartbeater.cc:511] Master 127.27.161.62:35967 requested a full tablet report, sending...
I20260810 03:27:18.633141 10705 ts_manager.cc:194] Registered new tserver with Master: 123904e534fa4243b7bcb411d792ee29 (127.27.161.3:44791)
I20260810 03:27:18.641920 28292 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260810 03:27:18.648416 10988 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:27:18.646803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39024 (local address 127.27.161.62:35967)
0810 03:27:18.646981 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0810 03:27:18.646987 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:27:18.647520 (+ 533us) server_negotiation.cc:407] Connection header received
0810 03:27:18.647709 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:27:18.647713 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:27:18.647774 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:27:18.647849 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:27:18.648118 (+ 269us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39024: BlockingRecv error: recv got EOF from 127.0.0.1:39024 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
I20260810 03:27:18.649282 28292 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iBH5/build/asan/bin/kudu pid 10793
I20260810 03:27:18.653260 11398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:27:18.650683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35464 (local address 127.27.161.1:42495)
0810 03:27:18.651185 (+ 502us) server_negotiation.cc:206] Beginning negotiation
0810 03:27:18.651193 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 03:27:18.651222 (+ 29us) server_negotiation.cc:407] Connection header received
0810 03:27:18.651285 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:27:18.651291 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:27:18.651514 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:27:18.651649 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:27:18.652572 (+ 923us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:27:18.652579 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:27:18.652592 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:27:18.652604 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:27:18.652651 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:27:18.652668 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:27:18.652772 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0810 03:27:18.652851 (+ 79us) server_negotiation.cc:299] Negotiation successful
0810 03:27:18.652948 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":336,"thread_start_us":212,"threads_started":1}
I20260810 03:27:18.654838 10979 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:27:18.736652 10985 heartbeater.cc:503] Master 127.27.161.62:35967 was elected leader, sending a full tablet report...
I20260810 03:27:18.739135 28292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iBH5/build/asan/bin/kudu with pid 10793
I20260810 03:27:18.767710 28292 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iBH5/build/asan/bin/kudu pid 10992
I20260810 03:27:18.771430 11401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:27:18.768667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47906 (local address 127.27.161.2:42919)
0810 03:27:18.769165 (+ 498us) server_negotiation.cc:206] Beginning negotiation
0810 03:27:18.769181 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 03:27:18.769209 (+ 28us) server_negotiation.cc:407] Connection header received
0810 03:27:18.769273 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:27:18.769279 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:27:18.769493 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:27:18.769663 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:27:18.770706 (+ 1043us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:27:18.770713 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:27:18.770725 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:27:18.770746 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:27:18.770789 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:27:18.770803 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:27:18.770931 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0810 03:27:18.771049 (+ 118us) server_negotiation.cc:299] Negotiation successful
0810 03:27:18.771156 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":359,"thread_start_us":223,"threads_started":1}
I20260810 03:27:18.772831 11181 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:27:18.853644 28292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iBH5/build/asan/bin/kudu with pid 10992
I20260810 03:27:18.877218 28292 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iBH5/build/asan/bin/kudu pid 11190
I20260810 03:27:18.880987 11406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:27:18.878123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36954 (local address 127.27.161.3:44791)
0810 03:27:18.878725 (+ 602us) server_negotiation.cc:206] Beginning negotiation
0810 03:27:18.878733 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 03:27:18.878763 (+ 30us) server_negotiation.cc:407] Connection header received
0810 03:27:18.878829 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:27:18.878835 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:27:18.879054 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:27:18.879204 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:27:18.880274 (+ 1070us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:27:18.880280 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:27:18.880291 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:27:18.880303 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:27:18.880341 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:27:18.880354 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:27:18.880517 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0810 03:27:18.880587 (+ 70us) server_negotiation.cc:299] Negotiation successful
0810 03:27:18.880712 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":441,"thread_start_us":265,"threads_started":1}
I20260810 03:27:18.882252 11372 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:27:18.963587 28292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iBH5/build/asan/bin/kudu with pid 11190
I20260810 03:27:18.988934 28292 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iBH5/build/asan/bin/kudu pid 10655
I20260810 03:27:18.990180 10753 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:27:19.094233 28292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iBH5/build/asan/bin/kudu with pid 10655
2026-08-10T03:27:19Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2568 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786332336319544-28292-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 03:27:19 dist-test-slave-vpvm krb5kdc[11416](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 03:27:19 dist-test-slave-vpvm krb5kdc[11416](info): set up 2 sockets
Aug 10 03:27:19 dist-test-slave-vpvm krb5kdc[11416](info): commencing operation
krb5kdc: starting...
W20260810 03:27:20.351778 28292 mini_kdc.cc:121] Time spent starting KDC: real 1.209s user 0.007s sys 0.046s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786332336319544-28292-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786332336319544-28292-0/krb5kdc/test-user.keytab.
Aug 10 03:27:20 dist-test-slave-vpvm krb5kdc[11416](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332440, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T03:27:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T03:27:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.161.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.161.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.161.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.62.keytab.
Entry for principal kudu/127.27.161.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.62.keytab.
I20260810 03:27:20.659904 28292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iBH5/build/asan/bin/kudu
/tmp/dist-test-task24iBH5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786332336319544-28292-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786332336319544-28292-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786332336319544-28292-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786332336319544-28292-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.161.62:34237
--webserver_interface=127.27.161.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.62.keytab
--principal=kudu/127.27.161.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.161.20:46413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.161.62:34237
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786332336319544-28292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786332336319544-28292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786332336319544-28292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:27:21.001899 11440 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:27:21.002214 11440 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:27:21.002336 11440 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:27:21.009475 11440 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:27:21.009586 11440 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:27:21.009657 11440 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:27:21.009735 11440 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:27:21.009795 11440 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:27:21.019802 11440 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.161.20:46413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786332336319544-28292-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786332336319544-28292-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.161.62:34237
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.62.keytab
--principal=kudu/127.27.161.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.161.62:34237
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786332336319544-28292-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.161.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786332336319544-28292-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:27:21.021165 11440 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:27:21.022451 11440 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:27:21.033810 11446 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:27:21.034189 11447 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:27:21.034531 11450 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:27:21.035099 11440 server_base.cc:1034] running on GCE node
I20260810 03:27:21.035879 11440 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:27:21.037343 11440 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:27:21.038542 11440 hybrid_clock.cc:648] HybridClock initialized: now 1786332441038501 us; error 52 us; skew 500 ppm
I20260810 03:27:21.038883 11440 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:27:21 dist-test-slave-vpvm krb5kdc[11416](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332441, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.161.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:27:21.041970 11440 init.cc:377] Logged in from keytab as kudu/127.27.161.62@KRBTEST.COM (short username kudu)
I20260810 03:27:21.043567 11440 webserver.cc:460] Webserver started at http://127.27.161.62:44115/ using document root <none> and password file <none>
I20260810 03:27:21.044242 11440 fs_manager.cc:359] Metadata directory not provided
I20260810 03:27:21.044344 11440 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:27:21.044636 11440 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:27:21.047724 11440 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786332336319544-28292-0/minicluster-data/master-0/data/instance:
uuid: "139256c1cf604da7a66995a9e9fb9442"
format_stamp: "Formatted at 2026-08-10 03:27:21 on dist-test-slave-vpvm"
server_key: "7340bc52931f84cb6b985e6576721f23"
server_key_iv: "85825ec28e42d20659fd241151030b1f"
server_key_version: "encryptionkey@0"
I20260810 03:27:21.048471 11440 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786332336319544-28292-0/minicluster-data/master-0/wal/instance:
uuid: "139256c1cf604da7a66995a9e9fb9442"
format_stamp: "Formatted at 2026-08-10 03:27:21 on dist-test-slave-vpvm"
server_key: "7340bc52931f84cb6b985e6576721f23"
server_key_iv: "85825ec28e42d20659fd241151030b1f"
server_key_version: "encryptionkey@0"
I20260810 03:27:21.053807 11440 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260810 03:27:21.057390 11461 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:27:21.058774 11440 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 03:27:21.058928 11440 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786332336319544-28292-0/minicluster-data/master-0/data,/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786332336319544-28292-0/minicluster-data/master-0/wal
uuid: "139256c1cf604da7a66995a9e9fb9442"
format_stamp: "Formatted at 2026-08-10 03:27:21 on dist-test-slave-vpvm"
server_key: "7340bc52931f84cb6b985e6576721f23"
server_key_iv: "85825ec28e42d20659fd241151030b1f"
server_key_version: "encryptionkey@0"
I20260810 03:27:21.059110 11440 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786332336319544-28292-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786332336319544-28292-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786332336319544-28292-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:27:21.094108 11440 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:27:21.098374 11440 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:27:21.098628 11440 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:27:21.116412 11440 rpc_server.cc:307] RPC server started. Bound to: 127.27.161.62:34237
I20260810 03:27:21.116451 11546 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.161.62:34237 every 8 connection(s)
I20260810 03:27:21.117813 11440 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786332336319544-28292-0/minicluster-data/master-0/data/info.pb
I20260810 03:27:21.119962 28292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iBH5/build/asan/bin/kudu as pid 11440
I20260810 03:27:21.120193 28292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786332336319544-28292-0/minicluster-data/master-0/wal/instance
I20260810 03:27:21.121042 28292 external_mini_cluster.cc:1428] Setting key 596a9678b935aee141b2744f5c583509
I20260810 03:27:21.122416 11548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:27:21.134443 11548 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 139256c1cf604da7a66995a9e9fb9442: Bootstrap starting.
Aug 10 03:27:21 dist-test-slave-vpvm krb5kdc[11416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332440, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.161.62@KRBTEST.COM
I20260810 03:27:21.138075 11548 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 139256c1cf604da7a66995a9e9fb9442: Neither blocks nor log segments found. Creating new log.
I20260810 03:27:21.139221 11548 log.cc:824] T 00000000000000000000000000000000 P 139256c1cf604da7a66995a9e9fb9442: Log is configured to *not* fsync() on all Append() calls
I20260810 03:27:21.142542 11548 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 139256c1cf604da7a66995a9e9fb9442: No bootstrap required, opened a new log
I20260810 03:27:21.142975 11552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:27:21.123316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48576 (local address 127.27.161.62:34237)
0810 03:27:21.124097 (+ 781us) server_negotiation.cc:206] Beginning negotiation
0810 03:27:21.124111 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 03:27:21.124153 (+ 42us) server_negotiation.cc:407] Connection header received
0810 03:27:21.125717 (+ 1564us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:27:21.125740 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:27:21.126188 (+ 448us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:27:21.126557 (+ 369us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:27:21.128079 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:27:21.129397 (+ 1318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:27:21.130671 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:27:21.131352 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:27:21.131542 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:27:21.135735 (+ 4193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:27:21.135743 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:27:21.135757 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:27:21.135802 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:27:21.138799 (+ 2997us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:27:21.139617 (+ 818us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:27:21.139624 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:27:21.139630 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:27:21.139836 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:27:21.140224 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:27:21.140229 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:27:21.140232 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:27:21.140889 (+ 657us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:27:21.141120 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0810 03:27:21.141550 (+ 430us) server_negotiation.cc:299] Negotiation successful
0810 03:27:21.141905 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":572,"thread_start_us":338,"threads_started":1}
I20260810 03:27:21.149979 11548 raft_consensus.cc:348] T 00000000000000000000000000000000 P 139256c1cf604da7a66995a9e9fb9442 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "139256c1cf604da7a66995a9e9fb9442" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 34237 } }
I20260810 03:27:21.150359 11548 raft_consensus.cc:374] T 00000000000000000000000000000000 P 139256c1cf604da7a66995a9e9fb9442 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:27:21.150461 11548 raft_consensus.cc:729] T 00000000000000000000000000000000 P 139256c1cf604da7a66995a9e9fb9442 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 139256c1cf604da7a66995a9e9fb9442, State: Initialized, Role: FOLLOWER
I20260810 03:27:21.151026 11548 consensus_queue.cc:260] T 00000000000000000000000000000000 P 139256c1cf604da7a66995a9e9fb9442 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "139256c1cf604da7a66995a9e9fb9442" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 34237 } }
I20260810 03:27:21.151238 11548 raft_consensus.cc:388] T 00000000000000000000000000000000 P 139256c1cf604da7a66995a9e9fb9442 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 03:27:21.151369 11548 raft_consensus.cc:482] T 00000000000000000000000000000000 P 139256c1cf604da7a66995a9e9fb9442 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 03:27:21.151542 11548 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 139256c1cf604da7a66995a9e9fb9442 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:27:21.153719 11548 raft_consensus.cc:504] T 00000000000000000000000000000000 P 139256c1cf604da7a66995a9e9fb9442 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "139256c1cf604da7a66995a9e9fb9442" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 34237 } }
I20260810 03:27:21.154241 11548 leader_election.cc:302] T 00000000000000000000000000000000 P 139256c1cf604da7a66995a9e9fb9442 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 139256c1cf604da7a66995a9e9fb9442; no voters:
I20260810 03:27:21.154911 11548 leader_election.cc:288] T 00000000000000000000000000000000 P 139256c1cf604da7a66995a9e9fb9442 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 03:27:21.155045 11555 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 139256c1cf604da7a66995a9e9fb9442 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:27:21.155661 11555 raft_consensus.cc:686] T 00000000000000000000000000000000 P 139256c1cf604da7a66995a9e9fb9442 [term 1 LEADER]: Becoming Leader. State: Replica: 139256c1cf604da7a66995a9e9fb9442, State: Running, Role: LEADER
I20260810 03:27:21.156263 11555 consensus_queue.cc:237] T 00000000000000000000000000000000 P 139256c1cf604da7a66995a9e9fb9442 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "139256c1cf604da7a66995a9e9fb9442" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 34237 } }
I20260810 03:27:21.157275 11548 sys_catalog.cc:564] T 00000000000000000000000000000000 P 139256c1cf604da7a66995a9e9fb9442 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:27:21.159245 11559 sys_catalog.cc:455] T 00000000000000000000000000000000 P 139256c1cf604da7a66995a9e9fb9442 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 139256c1cf604da7a66995a9e9fb9442. Latest consensus state: current_term: 1 leader_uuid: "139256c1cf604da7a66995a9e9fb9442" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "139256c1cf604da7a66995a9e9fb9442" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 34237 } } }
I20260810 03:27:21.159631 11559 sys_catalog.cc:458] T 00000000000000000000000000000000 P 139256c1cf604da7a66995a9e9fb9442 [sys.catalog]: This master's current role is: LEADER
I20260810 03:27:21.159395 11557 sys_catalog.cc:455] T 00000000000000000000000000000000 P 139256c1cf604da7a66995a9e9fb9442 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "139256c1cf604da7a66995a9e9fb9442" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "139256c1cf604da7a66995a9e9fb9442" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 34237 } } }
I20260810 03:27:21.159739 11557 sys_catalog.cc:458] T 00000000000000000000000000000000 P 139256c1cf604da7a66995a9e9fb9442 [sys.catalog]: This master's current role is: LEADER
I20260810 03:27:21.161377 11569 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 03:27:21.167522 11569 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 03:27:21.179034 11569 catalog_manager.cc:1347] Generated new cluster ID: 8b40e852d0df400a9feb9199dfb9edea
I20260810 03:27:21.179204 11569 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 03:27:21.194651 11569 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 03:27:21.196602 11569 catalog_manager.cc:1504] Loading token signing keys...
I20260810 03:27:21.206080 11569 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 139256c1cf604da7a66995a9e9fb9442: Generated new TSK 0
I20260810 03:27:21.207180 11569 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.161.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.161.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.161.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.1.keytab.
Entry for principal kudu/127.27.161.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.1.keytab.
I20260810 03:27:21.289527 28292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iBH5/build/asan/bin/kudu
/tmp/dist-test-task24iBH5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786332336319544-28292-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786332336319544-28292-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786332336319544-28292-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786332336319544-28292-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.161.1:0
--local_ip_for_outbound_sockets=127.27.161.1
--webserver_interface=127.27.161.1
--webserver_port=0
--tserver_master_addrs=127.27.161.62:34237
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.1.keytab
--principal=kudu/127.27.161.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.161.20:46413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786332336319544-28292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786332336319544-28292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786332336319544-28292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:27:21.640784 11592 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:27:21.641168 11592 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:27:21.641321 11592 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:27:21.648909 11592 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:27:21.649039 11592 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:27:21.649217 11592 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.161.1
I20260810 03:27:21.659379 11592 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.161.20:46413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786332336319544-28292-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786332336319544-28292-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.1.keytab
--principal=kudu/127.27.161.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.161.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786332336319544-28292-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.161.1
--webserver_port=0
--tserver_master_addrs=127.27.161.62:34237
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.161.1
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786332336319544-28292-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:27:21.660802 11592 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:27:21.662213 11592 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:27:21.675592 11607 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:27:21.675590 11601 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:27:21.676204 11600 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:27:21.676725 11592 server_base.cc:1034] running on GCE node
I20260810 03:27:21.677388 11592 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:27:21.678421 11592 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:27:21.679628 11592 hybrid_clock.cc:648] HybridClock initialized: now 1786332441679576 us; error 48 us; skew 500 ppm
I20260810 03:27:21.679946 11592 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:27:21 dist-test-slave-vpvm krb5kdc[11416](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332441, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.161.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:27:21.683053 11592 init.cc:377] Logged in from keytab as kudu/127.27.161.1@KRBTEST.COM (short username kudu)
I20260810 03:27:21.684501 11592 webserver.cc:460] Webserver started at http://127.27.161.1:40933/ using document root <none> and password file <none>
I20260810 03:27:21.685220 11592 fs_manager.cc:359] Metadata directory not provided
I20260810 03:27:21.685339 11592 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:27:21.685719 11592 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:27:21.688716 11592 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786332336319544-28292-0/minicluster-data/ts-0/data/instance:
uuid: "887a0f0e6d9b463d8157c41be58a0f01"
format_stamp: "Formatted at 2026-08-10 03:27:21 on dist-test-slave-vpvm"
server_key: "a597a300ba6771b3f68aa672a9de62dc"
server_key_iv: "9f512f4596584e23bcf87fca1ab27945"
server_key_version: "encryptionkey@0"
I20260810 03:27:21.689472 11592 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786332336319544-28292-0/minicluster-data/ts-0/wal/instance:
uuid: "887a0f0e6d9b463d8157c41be58a0f01"
format_stamp: "Formatted at 2026-08-10 03:27:21 on dist-test-slave-vpvm"
server_key: "a597a300ba6771b3f68aa672a9de62dc"
server_key_iv: "9f512f4596584e23bcf87fca1ab27945"
server_key_version: "encryptionkey@0"
I20260810 03:27:21.694720 11592 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260810 03:27:21.698172 11615 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:27:21.699478 11592 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260810 03:27:21.699647 11592 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786332336319544-28292-0/minicluster-data/ts-0/data,/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786332336319544-28292-0/minicluster-data/ts-0/wal
uuid: "887a0f0e6d9b463d8157c41be58a0f01"
format_stamp: "Formatted at 2026-08-10 03:27:21 on dist-test-slave-vpvm"
server_key: "a597a300ba6771b3f68aa672a9de62dc"
server_key_iv: "9f512f4596584e23bcf87fca1ab27945"
server_key_version: "encryptionkey@0"
I20260810 03:27:21.699841 11592 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786332336319544-28292-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786332336319544-28292-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786332336319544-28292-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:27:21.721174 11592 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:27:21.725543 11592 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:27:21.725834 11592 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:27:21.726851 11592 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:27:21.728258 11592 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:27:21.728338 11592 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:27:21.728469 11592 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:27:21.728524 11592 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:27:21.759764 11592 rpc_server.cc:307] RPC server started. Bound to: 127.27.161.1:37895
I20260810 03:27:21.759814 11799 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.161.1:37895 every 8 connection(s)
I20260810 03:27:21.761047 11592 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786332336319544-28292-0/minicluster-data/ts-0/data/info.pb
I20260810 03:27:21.771296 28292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iBH5/build/asan/bin/kudu as pid 11592
I20260810 03:27:21.771522 28292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786332336319544-28292-0/minicluster-data/ts-0/wal/instance
I20260810 03:27:21.772224 28292 external_mini_cluster.cc:1428] Setting key 8fbd892a904d5b99dca08c5883f448f6
Aug 10 03:27:21 dist-test-slave-vpvm krb5kdc[11416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332441, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.161.1@KRBTEST.COM for kudu/127.27.161.62@KRBTEST.COM
I20260810 03:27:21.780654 11804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:27:21.763674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.161.1:38653 (local address 127.27.161.62:34237)
0810 03:27:21.764133 (+ 459us) server_negotiation.cc:206] Beginning negotiation
0810 03:27:21.764139 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:27:21.765106 (+ 967us) server_negotiation.cc:407] Connection header received
0810 03:27:21.766964 (+ 1858us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:27:21.766969 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:27:21.767046 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:27:21.767135 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:27:21.768519 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:27:21.769220 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:27:21.770870 (+ 1650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:27:21.771468 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:27:21.771631 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:27:21.775189 (+ 3558us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:27:21.775195 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:27:21.775198 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:27:21.775231 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:27:21.777546 (+ 2315us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:27:21.778338 (+ 792us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:27:21.778343 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:27:21.778345 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:27:21.778468 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:27:21.778890 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:27:21.778893 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:27:21.778895 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:27:21.779166 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:27:21.779299 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0810 03:27:21.780034 (+ 735us) server_negotiation.cc:299] Negotiation successful
0810 03:27:21.780274 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":320,"thread_start_us":206,"threads_started":1}
I20260810 03:27:21.781606 11806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:27:21.764231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:34237 (local address 127.27.161.1:38653)
0810 03:27:21.764945 (+ 714us) negotiation.cc:107] Waiting for socket to connect
0810 03:27:21.764989 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0810 03:27:21.766707 (+ 1718us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:27:21.767331 (+ 624us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:27:21.767339 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:27:21.767823 (+ 484us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:27:21.768381 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:27:21.768399 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:27:21.769358 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:27:21.769361 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:27:21.770693 (+ 1332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:27:21.770707 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:27:21.771705 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:27:21.771712 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:27:21.771913 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:27:21.772584 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:27:21.772614 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:27:21.774973 (+ 2359us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:27:21.777692 (+ 2719us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:27:21.777702 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:27:21.777721 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:27:21.778214 (+ 493us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:27:21.778575 (+ 361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:27:21.778579 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:27:21.778582 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:27:21.778772 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:27:21.779294 (+ 522us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:27:21.779303 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:27:21.779642 (+ 339us) client_negotiation.cc:769] Sending connection context
0810 03:27:21.779884 (+ 242us) client_negotiation.cc:240] Negotiation successful
0810 03:27:21.780483 (+ 599us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":269,"threads_started":1}
I20260810 03:27:21.783103 11802 heartbeater.cc:347] Connected to a master server at 127.27.161.62:34237
I20260810 03:27:21.783394 11802 heartbeater.cc:464] Registering TS with master...
I20260810 03:27:21.784137 11802 heartbeater.cc:511] Master 127.27.161.62:34237 requested a full tablet report, sending...
I20260810 03:27:21.786237 11488 ts_manager.cc:194] Registered new tserver with Master: 887a0f0e6d9b463d8157c41be58a0f01 (127.27.161.1:37895)
I20260810 03:27:21.787672 11488 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.161.1@KRBTEST.COM'} at 127.27.161.1:38653
WARNING: no policy specified for kudu/127.27.161.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.161.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.161.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.2.keytab.
Entry for principal kudu/127.27.161.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.2.keytab.
I20260810 03:27:21.838449 28292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iBH5/build/asan/bin/kudu
/tmp/dist-test-task24iBH5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786332336319544-28292-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786332336319544-28292-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786332336319544-28292-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786332336319544-28292-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.161.2:0
--local_ip_for_outbound_sockets=127.27.161.2
--webserver_interface=127.27.161.2
--webserver_port=0
--tserver_master_addrs=127.27.161.62:34237
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.2.keytab
--principal=kudu/127.27.161.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.161.20:46413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786332336319544-28292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786332336319544-28292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786332336319544-28292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:27:22.197867 11809 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:27:22.198217 11809 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:27:22.198357 11809 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:27:22.205837 11809 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:27:22.205981 11809 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:27:22.206146 11809 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.161.2
I20260810 03:27:22.216244 11809 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.161.20:46413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786332336319544-28292-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786332336319544-28292-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.2.keytab
--principal=kudu/127.27.161.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.161.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786332336319544-28292-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.161.2
--webserver_port=0
--tserver_master_addrs=127.27.161.62:34237
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.161.2
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786332336319544-28292-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:27:22.217613 11809 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:27:22.219023 11809 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:27:22.232831 11815 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:27:22.232816 11817 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:27:22.233048 11822 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:27:22.233879 11809 server_base.cc:1034] running on GCE node
I20260810 03:27:22.234601 11809 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:27:22.235675 11809 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:27:22.236877 11809 hybrid_clock.cc:648] HybridClock initialized: now 1786332442236814 us; error 54 us; skew 500 ppm
I20260810 03:27:22.237205 11809 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:27:22 dist-test-slave-vpvm krb5kdc[11416](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332442, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.161.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:27:22.240446 11809 init.cc:377] Logged in from keytab as kudu/127.27.161.2@KRBTEST.COM (short username kudu)
I20260810 03:27:22.241930 11809 webserver.cc:460] Webserver started at http://127.27.161.2:33845/ using document root <none> and password file <none>
I20260810 03:27:22.242723 11809 fs_manager.cc:359] Metadata directory not provided
I20260810 03:27:22.242839 11809 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:27:22.243148 11809 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:27:22.246196 11809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786332336319544-28292-0/minicluster-data/ts-1/data/instance:
uuid: "faccd975f846476abfce2f5f5afd08b9"
format_stamp: "Formatted at 2026-08-10 03:27:22 on dist-test-slave-vpvm"
server_key: "cf15529d98de6e129a07b8ffcc6b01de"
server_key_iv: "c4969177ce2117343ff2d29e1df87e9d"
server_key_version: "encryptionkey@0"
I20260810 03:27:22.246949 11809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786332336319544-28292-0/minicluster-data/ts-1/wal/instance:
uuid: "faccd975f846476abfce2f5f5afd08b9"
format_stamp: "Formatted at 2026-08-10 03:27:22 on dist-test-slave-vpvm"
server_key: "cf15529d98de6e129a07b8ffcc6b01de"
server_key_iv: "c4969177ce2117343ff2d29e1df87e9d"
server_key_version: "encryptionkey@0"
I20260810 03:27:22.252952 11809 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.003s
I20260810 03:27:22.256758 11832 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:27:22.258201 11809 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 03:27:22.258360 11809 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786332336319544-28292-0/minicluster-data/ts-1/data,/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786332336319544-28292-0/minicluster-data/ts-1/wal
uuid: "faccd975f846476abfce2f5f5afd08b9"
format_stamp: "Formatted at 2026-08-10 03:27:22 on dist-test-slave-vpvm"
server_key: "cf15529d98de6e129a07b8ffcc6b01de"
server_key_iv: "c4969177ce2117343ff2d29e1df87e9d"
server_key_version: "encryptionkey@0"
I20260810 03:27:22.258558 11809 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786332336319544-28292-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786332336319544-28292-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786332336319544-28292-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:27:22.269212 11809 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:27:22.273350 11809 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:27:22.273651 11809 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:27:22.274730 11809 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:27:22.276132 11809 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:27:22.276208 11809 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:27:22.276340 11809 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:27:22.276391 11809 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:27:22.307142 11809 rpc_server.cc:307] RPC server started. Bound to: 127.27.161.2:35645
I20260810 03:27:22.307190 12006 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.161.2:35645 every 8 connection(s)
I20260810 03:27:22.308507 11809 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786332336319544-28292-0/minicluster-data/ts-1/data/info.pb
I20260810 03:27:22.311280 28292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iBH5/build/asan/bin/kudu as pid 11809
I20260810 03:27:22.311481 28292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786332336319544-28292-0/minicluster-data/ts-1/wal/instance
I20260810 03:27:22.312036 28292 external_mini_cluster.cc:1428] Setting key e53f78b7b2f44438b02d92d5e6412bf4
Aug 10 03:27:22 dist-test-slave-vpvm krb5kdc[11416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332442, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.161.2@KRBTEST.COM for kudu/127.27.161.62@KRBTEST.COM
I20260810 03:27:22.331709 12011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:27:22.311308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.161.2:51939 (local address 127.27.161.62:34237)
0810 03:27:22.311883 (+ 575us) server_negotiation.cc:206] Beginning negotiation
0810 03:27:22.311889 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:27:22.312973 (+ 1084us) server_negotiation.cc:407] Connection header received
0810 03:27:22.315141 (+ 2168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:27:22.315148 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:27:22.315240 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:27:22.315356 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:27:22.317129 (+ 1773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:27:22.318135 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:27:22.319779 (+ 1644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:27:22.320328 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:27:22.320444 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:27:22.324222 (+ 3778us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:27:22.324229 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:27:22.324234 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:27:22.324277 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:27:22.327551 (+ 3274us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:27:22.328621 (+ 1070us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:27:22.328628 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:27:22.328631 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:27:22.328818 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:27:22.329407 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:27:22.329412 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:27:22.329415 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:27:22.329787 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:27:22.329911 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0810 03:27:22.331028 (+ 1117us) server_negotiation.cc:299] Negotiation successful
0810 03:27:22.331288 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":435,"thread_start_us":302,"threads_started":1}
I20260810 03:27:22.334434 12012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:27:22.311896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:34237 (local address 127.27.161.2:51939)
0810 03:27:22.312792 (+ 896us) negotiation.cc:107] Waiting for socket to connect
0810 03:27:22.312852 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0810 03:27:22.314848 (+ 1996us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:27:22.315688 (+ 840us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:27:22.315698 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:27:22.316271 (+ 573us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:27:22.316919 (+ 648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:27:22.316938 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:27:22.318311 (+ 1373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:27:22.318315 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:27:22.319621 (+ 1306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:27:22.319635 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:27:22.320533 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:27:22.320538 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:27:22.320769 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:27:22.321436 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:27:22.321459 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:27:22.323949 (+ 2490us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:27:22.327784 (+ 3835us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:27:22.327796 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:27:22.327817 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:27:22.328470 (+ 653us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:27:22.328975 (+ 505us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:27:22.328979 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:27:22.328983 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:27:22.329263 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:27:22.330053 (+ 790us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:27:22.330062 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:27:22.330546 (+ 484us) client_negotiation.cc:769] Sending connection context
0810 03:27:22.332299 (+ 1753us) client_negotiation.cc:240] Negotiation successful
0810 03:27:22.332938 (+ 639us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":589,"thread_start_us":355,"threads_started":1}
I20260810 03:27:22.339169 12007 heartbeater.cc:347] Connected to a master server at 127.27.161.62:34237
I20260810 03:27:22.339655 12007 heartbeater.cc:464] Registering TS with master...
I20260810 03:27:22.340723 12007 heartbeater.cc:511] Master 127.27.161.62:34237 requested a full tablet report, sending...
I20260810 03:27:22.342751 11488 ts_manager.cc:194] Registered new tserver with Master: faccd975f846476abfce2f5f5afd08b9 (127.27.161.2:35645)
I20260810 03:27:22.343983 11488 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.161.2@KRBTEST.COM'} at 127.27.161.2:51939
WARNING: no policy specified for kudu/127.27.161.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.161.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.161.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.3.keytab.
Entry for principal kudu/127.27.161.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.3.keytab.
I20260810 03:27:22.385265 28292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iBH5/build/asan/bin/kudu
/tmp/dist-test-task24iBH5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786332336319544-28292-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786332336319544-28292-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786332336319544-28292-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786332336319544-28292-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.161.3:0
--local_ip_for_outbound_sockets=127.27.161.3
--webserver_interface=127.27.161.3
--webserver_port=0
--tserver_master_addrs=127.27.161.62:34237
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.3.keytab
--principal=kudu/127.27.161.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.161.20:46413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786332336319544-28292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786332336319544-28292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786332336319544-28292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:27:22.735602 12017 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:27:22.735893 12017 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:27:22.735996 12017 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:27:22.743296 12017 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:27:22.743414 12017 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:27:22.743536 12017 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.161.3
I20260810 03:27:22.753301 12017 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.161.20:46413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786332336319544-28292-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786332336319544-28292-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.3.keytab
--principal=kudu/127.27.161.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.161.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786332336319544-28292-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.161.3
--webserver_port=0
--tserver_master_addrs=127.27.161.62:34237
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.161.3
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786332336319544-28292-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:27:22.754660 12017 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:27:22.755939 12017 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:27:22.770450 12026 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:27:22.770471 12025 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:27:22.770412 12029 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:27:22.771198 12017 server_base.cc:1034] running on GCE node
I20260810 03:27:22.771836 12017 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:27:22.772766 12017 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:27:22.773984 12017 hybrid_clock.cc:648] HybridClock initialized: now 1786332442773916 us; error 32 us; skew 500 ppm
I20260810 03:27:22.774303 12017 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:27:22 dist-test-slave-vpvm krb5kdc[11416](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332442, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.161.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:27:22.777326 12017 init.cc:377] Logged in from keytab as kudu/127.27.161.3@KRBTEST.COM (short username kudu)
I20260810 03:27:22.778882 12017 webserver.cc:460] Webserver started at http://127.27.161.3:45555/ using document root <none> and password file <none>
I20260810 03:27:22.779580 12017 fs_manager.cc:359] Metadata directory not provided
I20260810 03:27:22.779706 12017 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:27:22.779999 12017 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:27:22.782984 12017 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786332336319544-28292-0/minicluster-data/ts-2/data/instance:
uuid: "47c22b2ae3414607a2e1de8c2ec88e2a"
format_stamp: "Formatted at 2026-08-10 03:27:22 on dist-test-slave-vpvm"
server_key: "f9add785750776c5f96aec301ad21cb9"
server_key_iv: "9a14d79ba61064ac0172f66081254e25"
server_key_version: "encryptionkey@0"
I20260810 03:27:22.783834 12017 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786332336319544-28292-0/minicluster-data/ts-2/wal/instance:
uuid: "47c22b2ae3414607a2e1de8c2ec88e2a"
format_stamp: "Formatted at 2026-08-10 03:27:22 on dist-test-slave-vpvm"
server_key: "f9add785750776c5f96aec301ad21cb9"
server_key_iv: "9a14d79ba61064ac0172f66081254e25"
server_key_version: "encryptionkey@0"
I20260810 03:27:22.789369 12017 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260810 03:27:22.790551 11802 heartbeater.cc:503] Master 127.27.161.62:34237 was elected leader, sending a full tablet report...
I20260810 03:27:22.793159 12037 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:27:22.794668 12017 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 03:27:22.794844 12017 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786332336319544-28292-0/minicluster-data/ts-2/data,/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786332336319544-28292-0/minicluster-data/ts-2/wal
uuid: "47c22b2ae3414607a2e1de8c2ec88e2a"
format_stamp: "Formatted at 2026-08-10 03:27:22 on dist-test-slave-vpvm"
server_key: "f9add785750776c5f96aec301ad21cb9"
server_key_iv: "9a14d79ba61064ac0172f66081254e25"
server_key_version: "encryptionkey@0"
I20260810 03:27:22.795044 12017 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786332336319544-28292-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786332336319544-28292-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786332336319544-28292-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:27:22.810851 12017 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:27:22.814764 12017 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:27:22.815078 12017 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:27:22.816103 12017 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:27:22.817545 12017 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:27:22.817625 12017 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:27:22.817766 12017 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:27:22.817838 12017 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:27:22.849601 12017 rpc_server.cc:307] RPC server started. Bound to: 127.27.161.3:45121
I20260810 03:27:22.849659 12207 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.161.3:45121 every 8 connection(s)
I20260810 03:27:22.850992 12017 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786332336319544-28292-0/minicluster-data/ts-2/data/info.pb
I20260810 03:27:22.861292 28292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iBH5/build/asan/bin/kudu as pid 12017
I20260810 03:27:22.861470 28292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786332336319544-28292-0/minicluster-data/ts-2/wal/instance
I20260810 03:27:22.862037 28292 external_mini_cluster.cc:1428] Setting key d387fdaf5f2d5cefd340c61a30f83693
Aug 10 03:27:22 dist-test-slave-vpvm krb5kdc[11416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332442, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.161.3@KRBTEST.COM for kudu/127.27.161.62@KRBTEST.COM
I20260810 03:27:22.872503 12214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:27:22.853498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.161.3:45863 (local address 127.27.161.62:34237)
0810 03:27:22.853986 (+ 488us) server_negotiation.cc:206] Beginning negotiation
0810 03:27:22.853992 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:27:22.854848 (+ 856us) server_negotiation.cc:407] Connection header received
0810 03:27:22.856687 (+ 1839us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:27:22.856694 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:27:22.856777 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:27:22.856875 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:27:22.858569 (+ 1694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:27:22.859436 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:27:22.861078 (+ 1642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:27:22.861548 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:27:22.861657 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:27:22.865918 (+ 4261us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:27:22.865924 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:27:22.865927 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:27:22.865995 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:27:22.869450 (+ 3455us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:27:22.870280 (+ 830us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:27:22.870284 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:27:22.870287 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:27:22.870413 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:27:22.870787 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:27:22.870790 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:27:22.870792 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:27:22.871071 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:27:22.871194 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0810 03:27:22.871909 (+ 715us) server_negotiation.cc:299] Negotiation successful
0810 03:27:22.872114 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":351,"thread_start_us":231,"threads_started":1}
I20260810 03:27:22.873409 12216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:27:22.853934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:34237 (local address 127.27.161.3:45863)
0810 03:27:22.854686 (+ 752us) negotiation.cc:107] Waiting for socket to connect
0810 03:27:22.854730 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0810 03:27:22.856380 (+ 1650us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:27:22.857140 (+ 760us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:27:22.857149 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:27:22.857703 (+ 554us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:27:22.858390 (+ 687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:27:22.858409 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:27:22.859606 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:27:22.859610 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:27:22.860915 (+ 1305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:27:22.860933 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:27:22.861864 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:27:22.861894 (+ 30us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:27:22.862285 (+ 391us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:27:22.863120 (+ 835us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:27:22.863146 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:27:22.865701 (+ 2555us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:27:22.869607 (+ 3906us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:27:22.869616 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:27:22.869632 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:27:22.870170 (+ 538us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:27:22.870497 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:27:22.870500 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:27:22.870502 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:27:22.870702 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:27:22.871175 (+ 473us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:27:22.871180 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:27:22.871571 (+ 391us) client_negotiation.cc:769] Sending connection context
0810 03:27:22.871809 (+ 238us) client_negotiation.cc:240] Negotiation successful
0810 03:27:22.872238 (+ 429us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":272,"threads_started":1}
I20260810 03:27:22.874945 12209 heartbeater.cc:347] Connected to a master server at 127.27.161.62:34237
I20260810 03:27:22.875255 12209 heartbeater.cc:464] Registering TS with master...
I20260810 03:27:22.876092 12209 heartbeater.cc:511] Master 127.27.161.62:34237 requested a full tablet report, sending...
I20260810 03:27:22.877624 11488 ts_manager.cc:194] Registered new tserver with Master: 47c22b2ae3414607a2e1de8c2ec88e2a (127.27.161.3:45121)
I20260810 03:27:22.878551 11488 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.161.3@KRBTEST.COM'} at 127.27.161.3:45863
I20260810 03:27:22.890014 28292 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 03:27:22.908159 12214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:27:22.894657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48590 (local address 127.27.161.62:34237)
0810 03:27:22.894889 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0810 03:27:22.894897 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 03:27:22.895208 (+ 311us) server_negotiation.cc:407] Connection header received
0810 03:27:22.895394 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:27:22.895401 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:27:22.895498 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:27:22.895594 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:27:22.897177 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:27:22.897936 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:27:22.899394 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:27:22.899876 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:27:22.900006 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:27:22.902522 (+ 2516us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:27:22.902528 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:27:22.902532 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:27:22.902571 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:27:22.905220 (+ 2649us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:27:22.905914 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:27:22.905920 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:27:22.905922 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:27:22.906108 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:27:22.906518 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:27:22.906524 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:27:22.906528 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:27:22.906902 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:27:22.907149 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0810 03:27:22.907577 (+ 428us) server_negotiation.cc:299] Negotiation successful
0810 03:27:22.907817 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260810 03:27:22.912402 11488 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48590:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 03:27:22.919502 11488 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 03:27:22.944465 12229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:27:22.931605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.2:35645 (local address 127.0.0.1:60436)
0810 03:27:22.933901 (+ 2296us) negotiation.cc:107] Waiting for socket to connect
0810 03:27:22.933918 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 03:27:22.934053 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:27:22.935567 (+ 1514us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:27:22.935577 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:27:22.935613 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:27:22.936080 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:27:22.936102 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:27:22.938528 (+ 2426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:27:22.938535 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:27:22.941180 (+ 2645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:27:22.941199 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:27:22.943390 (+ 2191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:27:22.943397 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:27:22.943627 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:27:22.943690 (+ 63us) client_negotiation.cc:769] Sending connection context
0810 03:27:22.943810 (+ 120us) client_negotiation.cc:240] Negotiation successful
0810 03:27:22.944021 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2165,"thread_start_us":1030,"threads_started":1}
I20260810 03:27:22.945991 12238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:27:22.932186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60436 (local address 127.27.161.2:35645)
0810 03:27:22.934840 (+ 2654us) server_negotiation.cc:206] Beginning negotiation
0810 03:27:22.934850 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 03:27:22.934884 (+ 34us) server_negotiation.cc:407] Connection header received
0810 03:27:22.934994 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:27:22.935003 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:27:22.935295 (+ 292us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:27:22.936329 (+ 1034us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:27:22.936585 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:27:22.938340 (+ 1755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:27:22.941429 (+ 3089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:27:22.943079 (+ 1650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:27:22.945058 (+ 1979us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:27:22.945325 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0810 03:27:22.945406 (+ 81us) server_negotiation.cc:299] Negotiation successful
0810 03:27:22.945605 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2256,"thread_start_us":2096,"threads_started":1}
I20260810 03:27:22.950078 11904 tablet_service.cc:1467] Processing CreateTablet for tablet 934f113e63024011ada6b245997ee110 (DEFAULT_TABLE table=test-table [id=5fb882b5c1a14787bf39c9d7e4b165ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:27:22.951701 11904 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 934f113e63024011ada6b245997ee110. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:27:22.953248 12229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:27:22.934571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.3:45121 (local address 127.0.0.1:59386)
0810 03:27:22.944744 (+ 10173us) negotiation.cc:107] Waiting for socket to connect
0810 03:27:22.944762 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 03:27:22.944871 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:27:22.947149 (+ 2278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:27:22.947154 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:27:22.947182 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:27:22.947777 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:27:22.947787 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:27:22.949695 (+ 1908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:27:22.949701 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:27:22.951113 (+ 1412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:27:22.951140 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:27:22.952426 (+ 1286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:27:22.952432 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:27:22.952600 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:27:22.952635 (+ 35us) client_negotiation.cc:769] Sending connection context
0810 03:27:22.952722 (+ 87us) client_negotiation.cc:240] Negotiation successful
0810 03:27:22.952875 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10087,"mutex_wait_us":52,"thread_start_us":11008,"threads_started":1}
I20260810 03:27:22.953241 12240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:27:22.935057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59386 (local address 127.27.161.3:45121)
0810 03:27:22.946529 (+ 11472us) server_negotiation.cc:206] Beginning negotiation
0810 03:27:22.946538 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 03:27:22.946572 (+ 34us) server_negotiation.cc:407] Connection header received
0810 03:27:22.946681 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:27:22.946689 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:27:22.946938 (+ 249us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:27:22.947139 (+ 201us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:27:22.947945 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:27:22.949494 (+ 1549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:27:22.951285 (+ 1791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:27:22.952259 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:27:22.952388 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:27:22.952639 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0810 03:27:22.952806 (+ 167us) server_negotiation.cc:299] Negotiation successful
0810 03:27:22.952960 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11277,"thread_start_us":11536,"threads_started":1}
I20260810 03:27:22.954437 12230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:27:22.931419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.1:37895 (local address 127.0.0.1:58894)
0810 03:27:22.933071 (+ 1652us) negotiation.cc:107] Waiting for socket to connect
0810 03:27:22.933127 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0810 03:27:22.933328 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:27:22.947179 (+ 13851us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:27:22.947183 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:27:22.947202 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:27:22.947633 (+ 431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:27:22.947645 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:27:22.949572 (+ 1927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:27:22.949582 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:27:22.951692 (+ 2110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:27:22.951708 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:27:22.953582 (+ 1874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:27:22.953589 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:27:22.953778 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:27:22.953810 (+ 32us) client_negotiation.cc:769] Sending connection context
0810 03:27:22.953882 (+ 72us) client_negotiation.cc:240] Negotiation successful
0810 03:27:22.954035 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1372,"mutex_wait_us":23,"thread_start_us":2197,"threads_started":1}
I20260810 03:27:22.955592 12235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:27:22.931790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58894 (local address 127.27.161.1:37895)
0810 03:27:22.946670 (+ 14880us) server_negotiation.cc:206] Beginning negotiation
0810 03:27:22.946678 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 03:27:22.946706 (+ 28us) server_negotiation.cc:407] Connection header received
0810 03:27:22.946800 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:27:22.946808 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:27:22.946982 (+ 174us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:27:22.947450 (+ 468us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:27:22.947808 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:27:22.949414 (+ 1606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:27:22.951894 (+ 2480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:27:22.953320 (+ 1426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:27:22.954802 (+ 1482us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:27:22.955033 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0810 03:27:22.955110 (+ 77us) server_negotiation.cc:299] Negotiation successful
0810 03:27:22.955267 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14714,"thread_start_us":14539,"threads_started":1}
I20260810 03:27:22.957278 12114 tablet_service.cc:1467] Processing CreateTablet for tablet 934f113e63024011ada6b245997ee110 (DEFAULT_TABLE table=test-table [id=5fb882b5c1a14787bf39c9d7e4b165ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:27:22.958853 12114 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 934f113e63024011ada6b245997ee110. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:27:22.959331 11697 tablet_service.cc:1467] Processing CreateTablet for tablet 934f113e63024011ada6b245997ee110 (DEFAULT_TABLE table=test-table [id=5fb882b5c1a14787bf39c9d7e4b165ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:27:22.960794 11697 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 934f113e63024011ada6b245997ee110. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:27:22.965308 12242 tablet_bootstrap.cc:492] T 934f113e63024011ada6b245997ee110 P faccd975f846476abfce2f5f5afd08b9: Bootstrap starting.
I20260810 03:27:22.968448 12242 tablet_bootstrap.cc:654] T 934f113e63024011ada6b245997ee110 P faccd975f846476abfce2f5f5afd08b9: Neither blocks nor log segments found. Creating new log.
I20260810 03:27:22.969485 12242 log.cc:824] T 934f113e63024011ada6b245997ee110 P faccd975f846476abfce2f5f5afd08b9: Log is configured to *not* fsync() on all Append() calls
I20260810 03:27:22.973289 12242 tablet_bootstrap.cc:492] T 934f113e63024011ada6b245997ee110 P faccd975f846476abfce2f5f5afd08b9: No bootstrap required, opened a new log
I20260810 03:27:22.973716 12242 ts_tablet_manager.cc:1397] T 934f113e63024011ada6b245997ee110 P faccd975f846476abfce2f5f5afd08b9: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260810 03:27:22.974028 12244 tablet_bootstrap.cc:492] T 934f113e63024011ada6b245997ee110 P 47c22b2ae3414607a2e1de8c2ec88e2a: Bootstrap starting.
I20260810 03:27:22.975654 12245 tablet_bootstrap.cc:492] T 934f113e63024011ada6b245997ee110 P 887a0f0e6d9b463d8157c41be58a0f01: Bootstrap starting.
I20260810 03:27:22.977866 12244 tablet_bootstrap.cc:654] T 934f113e63024011ada6b245997ee110 P 47c22b2ae3414607a2e1de8c2ec88e2a: Neither blocks nor log segments found. Creating new log.
I20260810 03:27:22.978451 12245 tablet_bootstrap.cc:654] T 934f113e63024011ada6b245997ee110 P 887a0f0e6d9b463d8157c41be58a0f01: Neither blocks nor log segments found. Creating new log.
I20260810 03:27:22.979231 12244 log.cc:824] T 934f113e63024011ada6b245997ee110 P 47c22b2ae3414607a2e1de8c2ec88e2a: Log is configured to *not* fsync() on all Append() calls
I20260810 03:27:22.979446 12245 log.cc:824] T 934f113e63024011ada6b245997ee110 P 887a0f0e6d9b463d8157c41be58a0f01: Log is configured to *not* fsync() on all Append() calls
I20260810 03:27:22.982321 12244 tablet_bootstrap.cc:492] T 934f113e63024011ada6b245997ee110 P 47c22b2ae3414607a2e1de8c2ec88e2a: No bootstrap required, opened a new log
I20260810 03:27:22.982589 12244 ts_tablet_manager.cc:1397] T 934f113e63024011ada6b245997ee110 P 47c22b2ae3414607a2e1de8c2ec88e2a: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.003s
I20260810 03:27:22.983332 12245 tablet_bootstrap.cc:492] T 934f113e63024011ada6b245997ee110 P 887a0f0e6d9b463d8157c41be58a0f01: No bootstrap required, opened a new log
I20260810 03:27:22.983606 12245 ts_tablet_manager.cc:1397] T 934f113e63024011ada6b245997ee110 P 887a0f0e6d9b463d8157c41be58a0f01: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260810 03:27:22.984723 12242 raft_consensus.cc:348] T 934f113e63024011ada6b245997ee110 P faccd975f846476abfce2f5f5afd08b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "887a0f0e6d9b463d8157c41be58a0f01" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 37895 } } peers { permanent_uuid: "faccd975f846476abfce2f5f5afd08b9" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 35645 } } peers { permanent_uuid: "47c22b2ae3414607a2e1de8c2ec88e2a" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 45121 } }
I20260810 03:27:22.985203 12242 raft_consensus.cc:374] T 934f113e63024011ada6b245997ee110 P faccd975f846476abfce2f5f5afd08b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:27:22.985342 12242 raft_consensus.cc:729] T 934f113e63024011ada6b245997ee110 P faccd975f846476abfce2f5f5afd08b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: faccd975f846476abfce2f5f5afd08b9, State: Initialized, Role: FOLLOWER
I20260810 03:27:22.986014 12242 consensus_queue.cc:260] T 934f113e63024011ada6b245997ee110 P faccd975f846476abfce2f5f5afd08b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "887a0f0e6d9b463d8157c41be58a0f01" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 37895 } } peers { permanent_uuid: "faccd975f846476abfce2f5f5afd08b9" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 35645 } } peers { permanent_uuid: "47c22b2ae3414607a2e1de8c2ec88e2a" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 45121 } }
I20260810 03:27:22.990267 12244 raft_consensus.cc:348] T 934f113e63024011ada6b245997ee110 P 47c22b2ae3414607a2e1de8c2ec88e2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "887a0f0e6d9b463d8157c41be58a0f01" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 37895 } } peers { permanent_uuid: "faccd975f846476abfce2f5f5afd08b9" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 35645 } } peers { permanent_uuid: "47c22b2ae3414607a2e1de8c2ec88e2a" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 45121 } }
I20260810 03:27:22.990796 12244 raft_consensus.cc:374] T 934f113e63024011ada6b245997ee110 P 47c22b2ae3414607a2e1de8c2ec88e2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:27:22.990778 12007 heartbeater.cc:503] Master 127.27.161.62:34237 was elected leader, sending a full tablet report...
I20260810 03:27:22.990950 12244 raft_consensus.cc:729] T 934f113e63024011ada6b245997ee110 P 47c22b2ae3414607a2e1de8c2ec88e2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47c22b2ae3414607a2e1de8c2ec88e2a, State: Initialized, Role: FOLLOWER
I20260810 03:27:22.991546 12242 ts_tablet_manager.cc:1428] T 934f113e63024011ada6b245997ee110 P faccd975f846476abfce2f5f5afd08b9: Time spent starting tablet: real 0.018s user 0.013s sys 0.004s
I20260810 03:27:22.991739 12244 consensus_queue.cc:260] T 934f113e63024011ada6b245997ee110 P 47c22b2ae3414607a2e1de8c2ec88e2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "887a0f0e6d9b463d8157c41be58a0f01" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 37895 } } peers { permanent_uuid: "faccd975f846476abfce2f5f5afd08b9" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 35645 } } peers { permanent_uuid: "47c22b2ae3414607a2e1de8c2ec88e2a" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 45121 } }
I20260810 03:27:22.992871 12209 heartbeater.cc:503] Master 127.27.161.62:34237 was elected leader, sending a full tablet report...
I20260810 03:27:22.993566 12244 ts_tablet_manager.cc:1428] T 934f113e63024011ada6b245997ee110 P 47c22b2ae3414607a2e1de8c2ec88e2a: Time spent starting tablet: real 0.011s user 0.010s sys 0.000s
I20260810 03:27:22.994740 12245 raft_consensus.cc:348] T 934f113e63024011ada6b245997ee110 P 887a0f0e6d9b463d8157c41be58a0f01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "887a0f0e6d9b463d8157c41be58a0f01" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 37895 } } peers { permanent_uuid: "faccd975f846476abfce2f5f5afd08b9" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 35645 } } peers { permanent_uuid: "47c22b2ae3414607a2e1de8c2ec88e2a" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 45121 } }
I20260810 03:27:22.995263 12245 raft_consensus.cc:374] T 934f113e63024011ada6b245997ee110 P 887a0f0e6d9b463d8157c41be58a0f01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:27:22.995414 12245 raft_consensus.cc:729] T 934f113e63024011ada6b245997ee110 P 887a0f0e6d9b463d8157c41be58a0f01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 887a0f0e6d9b463d8157c41be58a0f01, State: Initialized, Role: FOLLOWER
I20260810 03:27:22.996038 12245 consensus_queue.cc:260] T 934f113e63024011ada6b245997ee110 P 887a0f0e6d9b463d8157c41be58a0f01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "887a0f0e6d9b463d8157c41be58a0f01" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 37895 } } peers { permanent_uuid: "faccd975f846476abfce2f5f5afd08b9" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 35645 } } peers { permanent_uuid: "47c22b2ae3414607a2e1de8c2ec88e2a" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 45121 } }
I20260810 03:27:22.998553 12245 ts_tablet_manager.cc:1428] T 934f113e63024011ada6b245997ee110 P 887a0f0e6d9b463d8157c41be58a0f01: Time spent starting tablet: real 0.015s user 0.011s sys 0.004s
W20260810 03:27:23.013459 11803 tablet.cc:2369] T 934f113e63024011ada6b245997ee110 P 887a0f0e6d9b463d8157c41be58a0f01: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 03:27:23.060765 12009 tablet.cc:2369] T 934f113e63024011ada6b245997ee110 P faccd975f846476abfce2f5f5afd08b9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 03:27:23.102658 12211 tablet.cc:2369] T 934f113e63024011ada6b245997ee110 P 47c22b2ae3414607a2e1de8c2ec88e2a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 03:27:23.160426 12250 raft_consensus.cc:482] T 934f113e63024011ada6b245997ee110 P faccd975f846476abfce2f5f5afd08b9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 03:27:23.160760 12250 raft_consensus.cc:504] T 934f113e63024011ada6b245997ee110 P faccd975f846476abfce2f5f5afd08b9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "887a0f0e6d9b463d8157c41be58a0f01" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 37895 } } peers { permanent_uuid: "faccd975f846476abfce2f5f5afd08b9" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 35645 } } peers { permanent_uuid: "47c22b2ae3414607a2e1de8c2ec88e2a" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 45121 } }
I20260810 03:27:23.162699 12250 leader_election.cc:288] T 934f113e63024011ada6b245997ee110 P faccd975f846476abfce2f5f5afd08b9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 887a0f0e6d9b463d8157c41be58a0f01 (127.27.161.1:37895), 47c22b2ae3414607a2e1de8c2ec88e2a (127.27.161.3:45121)
I20260810 03:27:23.170300 12240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:27:23.163410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.161.2:35817 (local address 127.27.161.3:45121)
0810 03:27:23.163629 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0810 03:27:23.163635 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:27:23.164311 (+ 676us) server_negotiation.cc:407] Connection header received
0810 03:27:23.164512 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:27:23.164518 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:27:23.164607 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:27:23.164720 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:27:23.165325 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:27:23.166179 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:27:23.168358 (+ 2179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:27:23.169249 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:27:23.169362 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:27:23.169419 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0810 03:27:23.169890 (+ 471us) server_negotiation.cc:299] Negotiation successful
0810 03:27:23.170044 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260810 03:27:23.170305 12235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:27:23.163158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.161.2:38365 (local address 127.27.161.1:37895)
0810 03:27:23.163451 (+ 293us) server_negotiation.cc:206] Beginning negotiation
0810 03:27:23.163458 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:27:23.164363 (+ 905us) server_negotiation.cc:407] Connection header received
0810 03:27:23.164564 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:27:23.164569 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:27:23.164664 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:27:23.164746 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:27:23.165571 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:27:23.166363 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:27:23.168316 (+ 1953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:27:23.169339 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:27:23.169489 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:27:23.169563 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0810 03:27:23.169918 (+ 355us) server_negotiation.cc:299] Negotiation successful
0810 03:27:23.170067 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260810 03:27:23.170469 12254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:27:23.163026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.3:45121 (local address 127.27.161.2:35817)
0810 03:27:23.164174 (+ 1148us) negotiation.cc:107] Waiting for socket to connect
0810 03:27:23.164194 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 03:27:23.164361 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:27:23.164755 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:27:23.164761 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:27:23.164790 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:27:23.165170 (+ 380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:27:23.165182 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:27:23.166317 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:27:23.166323 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:27:23.168171 (+ 1848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:27:23.168187 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:27:23.169594 (+ 1407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:27:23.169598 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:27:23.169727 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:27:23.169757 (+ 30us) client_negotiation.cc:769] Sending connection context
0810 03:27:23.169826 (+ 69us) client_negotiation.cc:240] Negotiation successful
0810 03:27:23.169986 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1013,"mutex_wait_us":83,"thread_start_us":669,"threads_started":1}
I20260810 03:27:23.170506 12256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:27:23.163098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.1:37895 (local address 127.27.161.2:38365)
0810 03:27:23.164252 (+ 1154us) negotiation.cc:107] Waiting for socket to connect
0810 03:27:23.164265 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 03:27:23.164424 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:27:23.164906 (+ 482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:27:23.164909 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:27:23.164967 (+ 58us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:27:23.165373 (+ 406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:27:23.165386 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:27:23.166504 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:27:23.166511 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:27:23.168165 (+ 1654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:27:23.168178 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:27:23.169589 (+ 1411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:27:23.169605 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:27:23.169755 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:27:23.169779 (+ 24us) client_negotiation.cc:769] Sending connection context
0810 03:27:23.169867 (+ 88us) client_negotiation.cc:240] Negotiation successful
0810 03:27:23.170029 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1057,"mutex_wait_us":86,"spinlock_wait_cycles":68224,"thread_start_us":779,"threads_started":1}
I20260810 03:27:23.171554 12142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "934f113e63024011ada6b245997ee110" candidate_uuid: "faccd975f846476abfce2f5f5afd08b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47c22b2ae3414607a2e1de8c2ec88e2a" is_pre_election: true
I20260810 03:27:23.171557 11730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "934f113e63024011ada6b245997ee110" candidate_uuid: "faccd975f846476abfce2f5f5afd08b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "887a0f0e6d9b463d8157c41be58a0f01" is_pre_election: true
I20260810 03:27:23.172041 12142 raft_consensus.cc:2454] T 934f113e63024011ada6b245997ee110 P 47c22b2ae3414607a2e1de8c2ec88e2a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate faccd975f846476abfce2f5f5afd08b9 in term 0.
I20260810 03:27:23.172027 11730 raft_consensus.cc:2454] T 934f113e63024011ada6b245997ee110 P 887a0f0e6d9b463d8157c41be58a0f01 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate faccd975f846476abfce2f5f5afd08b9 in term 0.
I20260810 03:27:23.172665 11840 leader_election.cc:302] T 934f113e63024011ada6b245997ee110 P faccd975f846476abfce2f5f5afd08b9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47c22b2ae3414607a2e1de8c2ec88e2a, faccd975f846476abfce2f5f5afd08b9; no voters:
I20260810 03:27:23.173033 12250 raft_consensus.cc:2781] T 934f113e63024011ada6b245997ee110 P faccd975f846476abfce2f5f5afd08b9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 03:27:23.173192 12250 raft_consensus.cc:482] T 934f113e63024011ada6b245997ee110 P faccd975f846476abfce2f5f5afd08b9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 03:27:23.173297 12250 raft_consensus.cc:3037] T 934f113e63024011ada6b245997ee110 P faccd975f846476abfce2f5f5afd08b9 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:27:23.175482 12250 raft_consensus.cc:504] T 934f113e63024011ada6b245997ee110 P faccd975f846476abfce2f5f5afd08b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "887a0f0e6d9b463d8157c41be58a0f01" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 37895 } } peers { permanent_uuid: "faccd975f846476abfce2f5f5afd08b9" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 35645 } } peers { permanent_uuid: "47c22b2ae3414607a2e1de8c2ec88e2a" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 45121 } }
I20260810 03:27:23.176043 12250 leader_election.cc:288] T 934f113e63024011ada6b245997ee110 P faccd975f846476abfce2f5f5afd08b9 [CANDIDATE]: Term 1 election: Requested vote from peers 887a0f0e6d9b463d8157c41be58a0f01 (127.27.161.1:37895), 47c22b2ae3414607a2e1de8c2ec88e2a (127.27.161.3:45121)
I20260810 03:27:23.176417 11730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "934f113e63024011ada6b245997ee110" candidate_uuid: "faccd975f846476abfce2f5f5afd08b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "887a0f0e6d9b463d8157c41be58a0f01"
I20260810 03:27:23.176513 12142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "934f113e63024011ada6b245997ee110" candidate_uuid: "faccd975f846476abfce2f5f5afd08b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47c22b2ae3414607a2e1de8c2ec88e2a"
I20260810 03:27:23.176599 11730 raft_consensus.cc:3037] T 934f113e63024011ada6b245997ee110 P 887a0f0e6d9b463d8157c41be58a0f01 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:27:23.176717 12142 raft_consensus.cc:3037] T 934f113e63024011ada6b245997ee110 P 47c22b2ae3414607a2e1de8c2ec88e2a [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:27:23.178858 11730 raft_consensus.cc:2454] T 934f113e63024011ada6b245997ee110 P 887a0f0e6d9b463d8157c41be58a0f01 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate faccd975f846476abfce2f5f5afd08b9 in term 1.
I20260810 03:27:23.178897 12142 raft_consensus.cc:2454] T 934f113e63024011ada6b245997ee110 P 47c22b2ae3414607a2e1de8c2ec88e2a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate faccd975f846476abfce2f5f5afd08b9 in term 1.
I20260810 03:27:23.179322 11840 leader_election.cc:302] T 934f113e63024011ada6b245997ee110 P faccd975f846476abfce2f5f5afd08b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 47c22b2ae3414607a2e1de8c2ec88e2a, 887a0f0e6d9b463d8157c41be58a0f01, faccd975f846476abfce2f5f5afd08b9; no voters:
I20260810 03:27:23.179648 12250 raft_consensus.cc:2781] T 934f113e63024011ada6b245997ee110 P faccd975f846476abfce2f5f5afd08b9 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:27:23.180115 12250 raft_consensus.cc:686] T 934f113e63024011ada6b245997ee110 P faccd975f846476abfce2f5f5afd08b9 [term 1 LEADER]: Becoming Leader. State: Replica: faccd975f846476abfce2f5f5afd08b9, State: Running, Role: LEADER
I20260810 03:27:23.180590 12250 consensus_queue.cc:237] T 934f113e63024011ada6b245997ee110 P faccd975f846476abfce2f5f5afd08b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "887a0f0e6d9b463d8157c41be58a0f01" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 37895 } } peers { permanent_uuid: "faccd975f846476abfce2f5f5afd08b9" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 35645 } } peers { permanent_uuid: "47c22b2ae3414607a2e1de8c2ec88e2a" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 45121 } }
I20260810 03:27:23.185614 11488 catalog_manager.cc:5495] T 934f113e63024011ada6b245997ee110 P faccd975f846476abfce2f5f5afd08b9 reported cstate change: term changed from 0 to 1, leader changed from <none> to faccd975f846476abfce2f5f5afd08b9 (127.27.161.2). New cstate: current_term: 1 leader_uuid: "faccd975f846476abfce2f5f5afd08b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "887a0f0e6d9b463d8157c41be58a0f01" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 37895 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "faccd975f846476abfce2f5f5afd08b9" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 35645 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "47c22b2ae3414607a2e1de8c2ec88e2a" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 45121 } health_report { overall_health: UNKNOWN } } }
I20260810 03:27:23.238947 12238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:27:23.233661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60448 (local address 127.27.161.2:35645)
0810 03:27:23.233889 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0810 03:27:23.233894 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:27:23.233909 (+ 15us) server_negotiation.cc:407] Connection header received
0810 03:27:23.234038 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:27:23.234043 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:27:23.234140 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:27:23.234278 (+ 138us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 03:27:23.234666 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:27:23.235392 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:27:23.236742 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:27:23.237160 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:27:23.237272 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:27:23.237761 (+ 489us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 03:27:23.238348 (+ 587us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 03:27:23.238496 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0810 03:27:23.238589 (+ 93us) server_negotiation.cc:299] Negotiation successful
0810 03:27:23.238682 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260810 03:27:23.247625 11730 raft_consensus.cc:1261] T 934f113e63024011ada6b245997ee110 P 887a0f0e6d9b463d8157c41be58a0f01 [term 1 FOLLOWER]: Refusing update from remote peer faccd975f846476abfce2f5f5afd08b9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:27:23.247627 12142 raft_consensus.cc:1261] T 934f113e63024011ada6b245997ee110 P 47c22b2ae3414607a2e1de8c2ec88e2a [term 1 FOLLOWER]: Refusing update from remote peer faccd975f846476abfce2f5f5afd08b9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:27:23.248703 12259 consensus_queue.cc:1035] T 934f113e63024011ada6b245997ee110 P faccd975f846476abfce2f5f5afd08b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "887a0f0e6d9b463d8157c41be58a0f01" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 37895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:27:23.249115 12250 consensus_queue.cc:1035] T 934f113e63024011ada6b245997ee110 P faccd975f846476abfce2f5f5afd08b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47c22b2ae3414607a2e1de8c2ec88e2a" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 45121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:27:23.275288 11488 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48590:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 03:27:23.275591 11488 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48590:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 03:27:23.279863 11488 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 139256c1cf604da7a66995a9e9fb9442: Sending DeleteTablet for 3 replicas of tablet 934f113e63024011ada6b245997ee110
I20260810 03:27:23.281399 11697 tablet_service.cc:1514] Processing DeleteTablet for tablet 934f113e63024011ada6b245997ee110 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:27:23 UTC) from {username='kudu', principal='kudu/127.27.161.62@KRBTEST.COM'} at 127.0.0.1:58894
I20260810 03:27:23.281425 11904 tablet_service.cc:1514] Processing DeleteTablet for tablet 934f113e63024011ada6b245997ee110 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:27:23 UTC) from {username='kudu', principal='kudu/127.27.161.62@KRBTEST.COM'} at 127.0.0.1:60436
I20260810 03:27:23.281484 12114 tablet_service.cc:1514] Processing DeleteTablet for tablet 934f113e63024011ada6b245997ee110 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:27:23 UTC) from {username='kudu', principal='kudu/127.27.161.62@KRBTEST.COM'} at 127.0.0.1:59386
I20260810 03:27:23.284477 28292 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iBH5/build/asan/bin/kudu pid 11592
I20260810 03:27:23.293843 12280 tablet_replica.cc:330] T 934f113e63024011ada6b245997ee110 P faccd975f846476abfce2f5f5afd08b9: stopping tablet replica
I20260810 03:27:23.294452 12280 raft_consensus.cc:2229] T 934f113e63024011ada6b245997ee110 P faccd975f846476abfce2f5f5afd08b9 [term 1 LEADER]: Raft consensus shutting down.
I20260810 03:27:23.295238 12280 raft_consensus.cc:2258] T 934f113e63024011ada6b245997ee110 P faccd975f846476abfce2f5f5afd08b9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:27:23.296069 12282 tablet_replica.cc:330] T 934f113e63024011ada6b245997ee110 P 887a0f0e6d9b463d8157c41be58a0f01: stopping tablet replica
I20260810 03:27:23.297187 12282 raft_consensus.cc:2229] T 934f113e63024011ada6b245997ee110 P 887a0f0e6d9b463d8157c41be58a0f01 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 03:27:23.298183 12280 ts_tablet_manager.cc:1905] T 934f113e63024011ada6b245997ee110 P faccd975f846476abfce2f5f5afd08b9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:27:23.298851 12282 raft_consensus.cc:2258] T 934f113e63024011ada6b245997ee110 P 887a0f0e6d9b463d8157c41be58a0f01 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:27:23.302152 12282 ts_tablet_manager.cc:1905] T 934f113e63024011ada6b245997ee110 P 887a0f0e6d9b463d8157c41be58a0f01: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:27:23.303362 12284 tablet_replica.cc:330] T 934f113e63024011ada6b245997ee110 P 47c22b2ae3414607a2e1de8c2ec88e2a: stopping tablet replica
I20260810 03:27:23.303889 12284 raft_consensus.cc:2229] T 934f113e63024011ada6b245997ee110 P 47c22b2ae3414607a2e1de8c2ec88e2a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 03:27:23.304510 12284 raft_consensus.cc:2258] T 934f113e63024011ada6b245997ee110 P 47c22b2ae3414607a2e1de8c2ec88e2a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:27:23.309885 12284 ts_tablet_manager.cc:1905] T 934f113e63024011ada6b245997ee110 P 47c22b2ae3414607a2e1de8c2ec88e2a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:27:23.310580 12282 ts_tablet_manager.cc:1918] T 934f113e63024011ada6b245997ee110 P 887a0f0e6d9b463d8157c41be58a0f01: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 03:27:23.310868 12282 log.cc:1196] T 934f113e63024011ada6b245997ee110 P 887a0f0e6d9b463d8157c41be58a0f01: Deleting WAL directory at /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786332336319544-28292-0/minicluster-data/ts-0/wal/wals/934f113e63024011ada6b245997ee110
I20260810 03:27:23.311529 12282 ts_tablet_manager.cc:1939] T 934f113e63024011ada6b245997ee110 P 887a0f0e6d9b463d8157c41be58a0f01: Deleting consensus metadata
Aug 10 03:27:23 dist-test-slave-vpvm krb5kdc[11416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332440, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.161.1@KRBTEST.COM
I20260810 03:27:23.314524 11464 catalog_manager.cc:4841] TS 887a0f0e6d9b463d8157c41be58a0f01 (127.27.161.1:37895): tablet 934f113e63024011ada6b245997ee110 (table test-table [id=5fb882b5c1a14787bf39c9d7e4b165ff]) successfully deleted
I20260810 03:27:23.317655 12284 ts_tablet_manager.cc:1918] T 934f113e63024011ada6b245997ee110 P 47c22b2ae3414607a2e1de8c2ec88e2a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 03:27:23.317978 12284 log.cc:1196] T 934f113e63024011ada6b245997ee110 P 47c22b2ae3414607a2e1de8c2ec88e2a: Deleting WAL directory at /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786332336319544-28292-0/minicluster-data/ts-2/wal/wals/934f113e63024011ada6b245997ee110
I20260810 03:27:23.318645 12284 ts_tablet_manager.cc:1939] T 934f113e63024011ada6b245997ee110 P 47c22b2ae3414607a2e1de8c2ec88e2a: Deleting consensus metadata
I20260810 03:27:23.320833 11467 catalog_manager.cc:4841] TS 47c22b2ae3414607a2e1de8c2ec88e2a (127.27.161.3:45121): tablet 934f113e63024011ada6b245997ee110 (table test-table [id=5fb882b5c1a14787bf39c9d7e4b165ff]) successfully deleted
I20260810 03:27:23.323146 12280 ts_tablet_manager.cc:1918] T 934f113e63024011ada6b245997ee110 P faccd975f846476abfce2f5f5afd08b9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 03:27:23.323436 12280 log.cc:1196] T 934f113e63024011ada6b245997ee110 P faccd975f846476abfce2f5f5afd08b9: Deleting WAL directory at /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786332336319544-28292-0/minicluster-data/ts-1/wal/wals/934f113e63024011ada6b245997ee110
I20260810 03:27:23.324208 12280 ts_tablet_manager.cc:1939] T 934f113e63024011ada6b245997ee110 P faccd975f846476abfce2f5f5afd08b9: Deleting consensus metadata
I20260810 03:27:23.325493 12235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:27:23.286086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58904 (local address 127.27.161.1:37895)
0810 03:27:23.286303 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0810 03:27:23.286311 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 03:27:23.294159 (+ 7848us) server_negotiation.cc:407] Connection header received
0810 03:27:23.294510 (+ 351us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:27:23.294518 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:27:23.294644 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:27:23.294732 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:27:23.296839 (+ 2107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:27:23.298090 (+ 1251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:27:23.300070 (+ 1980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:27:23.300796 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:27:23.300917 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:27:23.316594 (+ 15677us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:27:23.316613 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:27:23.316632 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:27:23.316682 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:27:23.320812 (+ 4130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:27:23.321865 (+ 1053us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:27:23.321877 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:27:23.321880 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:27:23.322118 (+ 238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:27:23.322588 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:27:23.322593 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:27:23.322596 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:27:23.323118 (+ 522us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:27:23.324608 (+ 1490us) server_negotiation.cc:1035] Waiting for connection context
0810 03:27:23.324832 (+ 224us) server_negotiation.cc:299] Negotiation successful
0810 03:27:23.325084 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260810 03:27:23.326296 11467 catalog_manager.cc:4841] TS faccd975f846476abfce2f5f5afd08b9 (127.27.161.2:35645): tablet 934f113e63024011ada6b245997ee110 (table test-table [id=5fb882b5c1a14787bf39c9d7e4b165ff]) successfully deleted
I20260810 03:27:23.327653 11795 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:27:23.442309 28292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iBH5/build/asan/bin/kudu with pid 11592
I20260810 03:27:23.468468 28292 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iBH5/build/asan/bin/kudu pid 11809
Aug 10 03:27:23 dist-test-slave-vpvm krb5kdc[11416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332440, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.161.2@KRBTEST.COM
I20260810 03:27:23.485440 12238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:27:23.469518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60452 (local address 127.27.161.2:35645)
0810 03:27:23.469716 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0810 03:27:23.469721 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:27:23.469736 (+ 15us) server_negotiation.cc:407] Connection header received
0810 03:27:23.469793 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:27:23.469796 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:27:23.469857 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:27:23.469972 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:27:23.471247 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:27:23.472074 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:27:23.473349 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:27:23.473801 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:27:23.473928 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:27:23.478237 (+ 4309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:27:23.478259 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:27:23.478282 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:27:23.478331 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:27:23.482432 (+ 4101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:27:23.483154 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:27:23.483165 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:27:23.483168 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:27:23.483347 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:27:23.483710 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:27:23.483717 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:27:23.483720 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:27:23.484233 (+ 513us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:27:23.484557 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0810 03:27:23.484796 (+ 239us) server_negotiation.cc:299] Negotiation successful
0810 03:27:23.485041 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260810 03:27:23.486244 11999 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:27:23.602027 28292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iBH5/build/asan/bin/kudu with pid 11809
I20260810 03:27:23.629251 28292 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iBH5/build/asan/bin/kudu pid 12017
Aug 10 03:27:23 dist-test-slave-vpvm krb5kdc[11416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332440, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.161.3@KRBTEST.COM
I20260810 03:27:23.644090 12240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:27:23.630173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59388 (local address 127.27.161.3:45121)
0810 03:27:23.630353 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0810 03:27:23.630358 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:27:23.630378 (+ 20us) server_negotiation.cc:407] Connection header received
0810 03:27:23.630503 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:27:23.630507 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:27:23.630582 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:27:23.630665 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:27:23.631893 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:27:23.632624 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:27:23.633887 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:27:23.634403 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:27:23.634497 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:27:23.638834 (+ 4337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:27:23.638851 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:27:23.638869 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:27:23.638916 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:27:23.641544 (+ 2628us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:27:23.642302 (+ 758us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:27:23.642310 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:27:23.642312 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:27:23.642480 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:27:23.642773 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:27:23.642776 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:27:23.642778 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:27:23.643111 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:27:23.643350 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0810 03:27:23.643572 (+ 222us) server_negotiation.cc:299] Negotiation successful
0810 03:27:23.643742 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260810 03:27:23.645056 12203 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:27:23.733390 28292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iBH5/build/asan/bin/kudu with pid 12017
I20260810 03:27:23.757570 28292 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iBH5/build/asan/bin/kudu pid 11440
I20260810 03:27:23.758872 11542 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:27:23.867745 28292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iBH5/build/asan/bin/kudu with pid 11440
2026-08-10T03:27:23Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (4775 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786332336319544-28292-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 03:27:23 dist-test-slave-vpvm krb5kdc[12295](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 03:27:23 dist-test-slave-vpvm krb5kdc[12295](info): set up 2 sockets
Aug 10 03:27:23 dist-test-slave-vpvm krb5kdc[12295](info): commencing operation
krb5kdc: starting...
W20260810 03:27:25.123973 28292 mini_kdc.cc:121] Time spent starting KDC: real 1.206s user 0.001s sys 0.047s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786332336319544-28292-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786332336319544-28292-0/krb5kdc/test-user.keytab.
Aug 10 03:27:25 dist-test-slave-vpvm krb5kdc[12295](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332445, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T03:27:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T03:27:25Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.27.161.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.27.161.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.27.161.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786332336319544-28292-0/krb5kdc/rangeradmin_127.27.161.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.27.161.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786332336319544-28292-0/krb5kdc/rangeradmin_127.27.161.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.27.161.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.27.161.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.27.161.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786332336319544-28292-0/krb5kdc/rangerlookup_127.27.161.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.27.161.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786332336319544-28292-0/krb5kdc/rangerlookup_127.27.161.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.27.161.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.27.161.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.27.161.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786332336319544-28292-0/krb5kdc/HTTP_127.27.161.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.27.161.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786332336319544-28292-0/krb5kdc/HTTP_127.27.161.20@KRBTEST.COM.keytab.
I20260810 03:27:25.564677 28292 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786332336319544-28292-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task24iBH5/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786332336319544-28292-0/minicluster-data/postgres -l logfile start
2026-08-10 03:27:28.071 UTC [12343] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-10 03:27:28.071 UTC [12343] LOG: listening on IPv4 address "127.27.161.20", port 37731
2026-08-10 03:27:28.074 UTC [12343] LOG: listening on Unix socket "/tmp/.s.PGSQL.37731"
2026-08-10 03:27:28.096 UTC [12348] LOG: database system was shut down at 2026-08-10 03:27:26 UTC
2026-08-10 03:27:28.100 UTC [12343] LOG: database system is ready to accept connections
I20260810 03:27:29.195907 28292 mini_postgres.cc:96] Postgres bound to 37731
2026-08-10 03:27:29.224 UTC [12363] FATAL: database "slave" does not exist
127.27.161.20:37731 - accepting connections
I20260810 03:27:29.224992 28292 mini_ranger.cc:161] Starting Ranger...
I20260810 03:27:29.261838 28292 mini_ranger.cc:84] Created miniranger Postgres user
I20260810 03:27:29.754638 28292 mini_ranger.cc:87] Created ranger Postgres database
I20260810 03:27:29.754813 28292 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786332336319544-28292-0/minicluster-data/ranger-admin
2026-08-10 03:27:30,177 [I] DB FLAVOR :POSTGRES
2026-08-10 03:27:30,178 [I] --------- Verifying Ranger DB connection ---------
2026-08-10 03:27:30,178 [I] Checking connection..
2026-08-10 03:27:30,178 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task24iBH5/build/asan/bin/postgresql.jar:/tmp/dist-test-task24iBH5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.161.20:37731/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-10 03:27:30,466 [I] Checking connection passed.
2026-08-10 03:27:30,467 [I] --------- Verifying version history table ---------
2026-08-10 03:27:30,467 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task24iBH5/build/asan/bin/postgresql.jar:/tmp/dist-test-task24iBH5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.161.20:37731/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-10 03:27:30,726 [I] Table x_db_version_h does not exist in database ranger
2026-08-10 03:27:30,726 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task24iBH5/build/asan/bin/postgresql.jar:/tmp/dist-test-task24iBH5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.161.20:37731/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-10 03:27:30,998 [I] Table x_db_version_h does not exist in database ranger
2026-08-10 03:27:30,998 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-10 03:27:30,998 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task24iBH5/build/asan/bin/postgresql.jar:/tmp/dist-test-task24iBH5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.161.20:37731/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task24iBH5/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-10 03:27:31.243 UTC [12460] WARNING: there is no transaction in progress
2026-08-10 03:27:31,252 [I] create_dbversion_catalog.sql file imported successfully
2026-08-10 03:27:31,252 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task24iBH5/build/asan/bin/postgresql.jar:/tmp/dist-test-task24iBH5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.161.20:37731/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-10 03:27:31,524 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-10 03:27:31,524 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-10 03:27:31,524 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task24iBH5/build/asan/bin/postgresql.jar:/tmp/dist-test-task24iBH5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.161.20:37731/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-10 03:27:31,785 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task24iBH5/build/asan/bin/postgresql.jar:/tmp/dist-test-task24iBH5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.161.20:37731/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-10 03:27:32,049 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task24iBH5/build/asan/bin/postgresql.jar:/tmp/dist-test-task24iBH5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.161.20:37731/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-vpvm.c.gcp-upstream.internal','N') ;"
2026-08-10 03:27:32,287 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task24iBH5/build/asan/bin/postgresql.jar:/tmp/dist-test-task24iBH5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.161.20:37731/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-10 03:27:32,551 [I] Table x_portal_user does not exist in database ranger
2026-08-10 03:27:32,552 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task24iBH5/build/asan/bin/postgresql.jar:/tmp/dist-test-task24iBH5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.161.20:37731/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-10 03:27:32,816 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-10 03:27:32,816 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task24iBH5/build/asan/bin/postgresql.jar:/tmp/dist-test-task24iBH5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.161.20:37731/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-10 03:27:33,077 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task24iBH5/build/asan/bin/postgresql.jar:/tmp/dist-test-task24iBH5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.161.20:37731/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-10 03:27:33,356 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-10 03:27:33,356 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task24iBH5/build/asan/bin/postgresql.jar:/tmp/dist-test-task24iBH5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.161.20:37731/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task24iBH5/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-10 03:27:34.328 UTC [12612] WARNING: there is no transaction in progress
2026-08-10 03:27:34.344 UTC [12612] WARNING: there is no transaction in progress
2026-08-10 03:27:34.359 UTC [12612] WARNING: there is no transaction in progress
2026-08-10 03:27:34.374 UTC [12612] WARNING: there is no transaction in progress
2026-08-10 03:27:34.390 UTC [12612] WARNING: there is no transaction in progress
2026-08-10 03:27:34.408 UTC [12612] WARNING: there is no transaction in progress
2026-08-10 03:27:34.468 UTC [12612] WARNING: there is no transaction in progress
2026-08-10 03:27:34.475 UTC [12612] WARNING: there is no transaction in progress
2026-08-10 03:27:34.489 UTC [12612] WARNING: there is no transaction in progress
2026-08-10 03:27:34.498 UTC [12612] WARNING: there is no transaction in progress
2026-08-10 03:27:34.508 UTC [12612] WARNING: there is no transaction in progress
2026-08-10 03:27:34.522 UTC [12612] WARNING: there is no transaction in progress
2026-08-10 03:27:34.531 UTC [12612] WARNING: there is no transaction in progress
2026-08-10 03:27:34.537 UTC [12612] WARNING: there is no transaction in progress
2026-08-10 03:27:34.543 UTC [12612] WARNING: there is no transaction in progress
2026-08-10 03:27:34.549 UTC [12612] WARNING: there is no transaction in progress
2026-08-10 03:27:35,154 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-10 03:27:35,154 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task24iBH5/build/asan/bin/postgresql.jar:/tmp/dist-test-task24iBH5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.161.20:37731/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-10 03:27:35,409 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-10 03:27:35,409 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task24iBH5/build/asan/bin/postgresql.jar:/tmp/dist-test-task24iBH5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.161.20:37731/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-10 03:27:35,680 [I] Table x_portal_user already exists in database 'ranger'
2026-08-10 03:27:35,680 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task24iBH5/build/asan/bin/postgresql.jar:/tmp/dist-test-task24iBH5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.161.20:37731/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-10 03:27:35,950 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-10 03:27:35,950 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task24iBH5/build/asan/bin/postgresql.jar:/tmp/dist-test-task24iBH5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.161.20:37731/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-10 03:27:36,191 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task24iBH5/build/asan/bin/postgresql.jar:/tmp/dist-test-task24iBH5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.161.20:37731/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-10 03:27:36,446 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task24iBH5/build/asan/bin/postgresql.jar:/tmp/dist-test-task24iBH5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.161.20:37731/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-vpvm.c.gcp-upstream.internal';"
2026-08-10 03:27:36,706 [I] CORE_DB_SCHEMA import status has been updated
2026-08-10 03:27:36,706 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task24iBH5/build/asan/bin/postgresql.jar:/tmp/dist-test-task24iBH5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.161.20:37731/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-10 03:27:36,943 [I] DB_PATCHES have already been applied
I20260810 03:27:36.949607 28292 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786332336319544-28292-0/minicluster-data/ranger-admin:/tmp/dist-test-task24iBH5/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task24iBH5/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task24iBH5/build/asan/bin/postgresql.jar:/tmp/dist-test-task24iBH5/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260810 03:27:36.949736 28292 mini_ranger.cc:193] Using host: 127.27.161.20
I20260810 03:27:36.973309 28292 mini_ranger.cc:227] Ranger admin URL: http://127.27.161.20:36621
Aug 10, 2026 3:27:37 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 10, 2026 3:27:37 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 10, 2026 3:27:37 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786332336319544-28292-0/minicluster-data/ranger-admin/ews/webapp
Aug 10, 2026 3:27:37 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786332336319544-28292-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 10, 2026 3:27:37 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786332336319544-28292-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 10, 2026 3:27:37 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786332336319544-28292-0/minicluster-data/ranger-admin/ews/webapp].
Aug 10, 2026 3:27:37 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260810 03:27:38.261313 28292 mini_ranger.cc:160] Time spent starting Ranger: real 9.036s user 0.002s sys 0.078s
Aug 10, 2026 3:27:58 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 10, 2026 3:27:58 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Aug 10, 2026 3:27:58 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 10, 2026 3:27:58 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 10, 2026 3:27:58 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 10, 2026 3:27:58 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 10, 2026 3:27:58 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 10, 2026 3:27:58 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 10, 2026 3:27:58 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 10, 2026 3:27:58 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 10, 2026 3:27:58 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 10, 2026 3:27:58 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 10, 2026 3:27:58 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 10, 2026 3:27:58 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 10, 2026 3:27:58 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 10, 2026 3:27:58 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 10, 2026 3:27:58 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 10, 2026 3:27:59 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260810 03:28:00.016500 28292 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.27.161.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.27.161.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.27.161.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786332336319544-28292-0/krb5kdc/rangerkms_127.27.161.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.27.161.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786332336319544-28292-0/krb5kdc/rangerkms_127.27.161.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.27.161.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.27.161.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.27.161.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786332336319544-28292-0/krb5kdc/HTTP_127.27.161.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.27.161.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786332336319544-28292-0/krb5kdc/HTTP_127.27.161.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 10 03:28:00 dist-test-slave-vpvm krb5kdc[12295](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332480, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260810 03:28:00.258349 28292 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260810 03:28:00.290783 28292 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260810 03:28:01.100649 28292 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260810 03:28:01.100802 28292 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786332336319544-28292-0/minicluster-data/ranger-kms
I20260810 03:28:01.100950 28292 mini_ranger_kms.cc:227] Using postgres at 127.27.161.20:37731
2026-08-10 03:28:01,614 [I] DB FLAVOR :POSTGRES
2026-08-10 03:28:01,614 [I] --------- Verifying Ranger DB connection ---------
2026-08-10 03:28:01,614 [I] Checking connection
2026-08-10 03:28:01,614 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task24iBH5/build/asan/bin/postgresql.jar:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786332336319544-28292-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.161.20:37731/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-10 03:28:01,891 [I] connection success
2026-08-10 03:28:01,892 [I] --------- Verifying Ranger DB tables ---------
2026-08-10 03:28:01,892 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-10 03:28:01,892 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task24iBH5/build/asan/bin/postgresql.jar:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786332336319544-28292-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.161.20:37731/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-10 03:28:02,159 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-10 03:28:02,159 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-10 03:28:02,159 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-10 03:28:02,159 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task24iBH5/build/asan/bin/postgresql.jar:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786332336319544-28292-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.161.20:37731/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786332336319544-28292-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-10 03:28:02,461 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260810 03:28:02.974830 28292 mini_ranger_kms.cc:310] Created kms service
I20260810 03:28:03.186776 28292 mini_ranger_kms.cc:326] Created kudu user
I20260810 03:28:03.231352 28292 mini_ranger_kms.cc:343] Created rangerkms user
Aug 10 03:28:03 dist-test-slave-vpvm krb5kdc[12295](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332480, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.27.161.20@KRBTEST.COM
I20260810 03:28:03.688855 28292 mini_ranger_kms.cc:384] Added ranger policy
I20260810 03:28:03.690346 28292 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786332336319544-28292-0/minicluster-data/ranger-kms:/tmp/dist-test-task24iBH5/build/asan/bin/postgresql.jar:/tmp/dist-test-task24iBH5/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task24iBH5/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260810 03:28:03.690419 28292 mini_ranger_kms.cc:240] Using host: 127.27.161.20
I20260810 03:28:03.726161 28292 mini_ranger_kms.cc:276] Ranger KMS PID: 12898
I20260810 03:28:03.726351 28292 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.27.161.20:55767
Aug 10, 2026 3:28:04 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 10, 2026 3:28:04 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 10, 2026 3:28:05 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786332336319544-28292-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 10, 2026 3:28:05 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786332336319544-28292-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 10, 2026 3:28:05 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786332336319544-28292-0/minicluster-data/ranger-kms/ews/webapp].
Aug 10, 2026 3:28:05 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Aug 10 03:28:06 dist-test-slave-vpvm krb5kdc[12295](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786332486, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.27.161.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:28:06.324016 28292 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 6.066s user 0.141s sys 0.434s
I20260810 03:28:06.324374 28292 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260810 03:28:06.324589 28292 mini_ranger_kms.cc:401] 127.27.161.20:55767/kms/v1/keys
Aug 10 03:28:06 dist-test-slave-vpvm krb5kdc[12295](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332486, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.27.161.20@KRBTEST.COM for HTTP/127.27.161.20@KRBTEST.COM
Aug 10 03:28:07 dist-test-slave-vpvm krb5kdc[12295](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332486, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.27.161.20@KRBTEST.COM for HTTP/127.27.161.20@KRBTEST.COM
Aug 10 03:28:09 dist-test-slave-vpvm krb5kdc[12295](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332489, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.27.161.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.161.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.161.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.62.keytab.
Entry for principal kudu/127.27.161.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.62.keytab.
I20260810 03:28:09.756978 28292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iBH5/build/asan/bin/kudu
/tmp/dist-test-task24iBH5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786332336319544-28292-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786332336319544-28292-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786332336319544-28292-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786332336319544-28292-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.161.62:32815
--webserver_interface=127.27.161.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.62.keytab
--principal=kudu/127.27.161.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.161.20:43399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.161.62:32815
--ranger_config_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786332336319544-28292-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.161.20:55767/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786332336319544-28292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786332336319544-28292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786332336319544-28292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:28:10.168023 12983 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:28:10.168349 12983 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:28:10.168511 12983 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:28:10.176465 12983 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:28:10.176597 12983 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:28:10.176679 12983 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:28:10.176759 12983 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:28:10.176816 12983 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:28:10.188064 12983 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.161.20:43399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786332336319544-28292-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786332336319544-28292-0/minicluster-data/master-0/wal
--ranger_kms_url=127.27.161.20:55767/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.27.161.62:32815
--ranger_config_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786332336319544-28292-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.62.keytab
--principal=kudu/127.27.161.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.161.62:32815
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786332336319544-28292-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.161.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786332336319544-28292-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:28:10.189455 12983 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:28:10.190918 12983 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:28:10.205111 12994 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:28:10.205111 12991 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:28:10.204828 12990 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:28:10.206081 12983 server_base.cc:1034] running on GCE node
I20260810 03:28:10.207055 12983 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:28:10.208683 12983 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:28:10.209910 12983 hybrid_clock.cc:648] HybridClock initialized: now 1786332490209855 us; error 49 us; skew 500 ppm
I20260810 03:28:10.210317 12983 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:28:10 dist-test-slave-vpvm krb5kdc[12295](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332490, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.161.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:28:10.216358 12983 init.cc:377] Logged in from keytab as kudu/127.27.161.62@KRBTEST.COM (short username kudu)
I20260810 03:28:10.218066 12983 webserver.cc:460] Webserver started at http://127.27.161.62:37639/ using document root <none> and password file <none>
I20260810 03:28:10.218806 12983 fs_manager.cc:359] Metadata directory not provided
I20260810 03:28:10.218926 12983 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:28:10.219236 12983 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 03:28:10 dist-test-slave-vpvm krb5kdc[12295](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332490, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.161.62@KRBTEST.COM for HTTP/127.27.161.20@KRBTEST.COM
I20260810 03:28:10.284096 12983 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786332336319544-28292-0/minicluster-data/master-0/data/instance:
uuid: "f0c134cae7974fed9683afccfc749107"
format_stamp: "Formatted at 2026-08-10 03:28:10 on dist-test-slave-vpvm"
server_key: "8dea3bd33d149f55610191b9a6a8499a"
server_key_iv: "bb32809cfd7de7d78c4ebbb377080121"
server_key_version: "kuduclusterkey@0"
I20260810 03:28:10.285463 12983 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786332336319544-28292-0/minicluster-data/master-0/wal/instance:
uuid: "f0c134cae7974fed9683afccfc749107"
format_stamp: "Formatted at 2026-08-10 03:28:10 on dist-test-slave-vpvm"
server_key: "8dea3bd33d149f55610191b9a6a8499a"
server_key_iv: "bb32809cfd7de7d78c4ebbb377080121"
server_key_version: "kuduclusterkey@0"
I20260810 03:28:10.293524 12983 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.001s
I20260810 03:28:10.314992 13002 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:28:10.317134 12983 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260810 03:28:10.317400 12983 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786332336319544-28292-0/minicluster-data/master-0/data,/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786332336319544-28292-0/minicluster-data/master-0/wal
uuid: "f0c134cae7974fed9683afccfc749107"
format_stamp: "Formatted at 2026-08-10 03:28:10 on dist-test-slave-vpvm"
server_key: "8dea3bd33d149f55610191b9a6a8499a"
server_key_iv: "bb32809cfd7de7d78c4ebbb377080121"
server_key_version: "kuduclusterkey@0"
I20260810 03:28:10.317662 12983 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786332336319544-28292-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786332336319544-28292-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786332336319544-28292-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:28:10.353183 12983 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:28:10.357378 12983 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:28:10.357687 12983 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:28:10.378229 12983 rpc_server.cc:307] RPC server started. Bound to: 127.27.161.62:32815
I20260810 03:28:10.378270 13085 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.161.62:32815 every 8 connection(s)
I20260810 03:28:10.379792 12983 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786332336319544-28292-0/minicluster-data/master-0/data/info.pb
I20260810 03:28:10.383854 13086 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:28:10.388696 28292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iBH5/build/asan/bin/kudu as pid 12983
I20260810 03:28:10.388906 28292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786332336319544-28292-0/minicluster-data/master-0/wal/instance
I20260810 03:28:10.389724 28292 external_mini_cluster.cc:1428] Setting key a7c011f9173eb57f4b2bbb938c8263b0
I20260810 03:28:10.396162 13086 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f0c134cae7974fed9683afccfc749107: Bootstrap starting.
I20260810 03:28:10.399718 13086 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f0c134cae7974fed9683afccfc749107: Neither blocks nor log segments found. Creating new log.
I20260810 03:28:10.400903 13086 log.cc:824] T 00000000000000000000000000000000 P f0c134cae7974fed9683afccfc749107: Log is configured to *not* fsync() on all Append() calls
Aug 10 03:28:10 dist-test-slave-vpvm krb5kdc[12295](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332489, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.161.62@KRBTEST.COM
I20260810 03:28:10.404253 13086 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f0c134cae7974fed9683afccfc749107: No bootstrap required, opened a new log
I20260810 03:28:10.411947 13090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:28:10.391992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58108 (local address 127.27.161.62:32815)
0810 03:28:10.392847 (+ 855us) server_negotiation.cc:206] Beginning negotiation
0810 03:28:10.392859 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 03:28:10.392903 (+ 44us) server_negotiation.cc:407] Connection header received
0810 03:28:10.394800 (+ 1897us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:28:10.394826 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:28:10.395237 (+ 411us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:28:10.395586 (+ 349us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:28:10.396947 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:28:10.398412 (+ 1465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:28:10.399640 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:28:10.400376 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:28:10.400551 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:28:10.404889 (+ 4338us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:28:10.404897 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:28:10.404913 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:28:10.404954 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:28:10.407971 (+ 3017us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:28:10.408643 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:28:10.408650 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:28:10.408656 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:28:10.408806 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:28:10.409187 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:28:10.409191 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:28:10.409193 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:28:10.409769 (+ 576us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:28:10.410101 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0810 03:28:10.410552 (+ 451us) server_negotiation.cc:299] Negotiation successful
0810 03:28:10.410950 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":639,"thread_start_us":402,"threads_started":1}
I20260810 03:28:10.412357 13086 raft_consensus.cc:348] T 00000000000000000000000000000000 P f0c134cae7974fed9683afccfc749107 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0c134cae7974fed9683afccfc749107" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 32815 } }
I20260810 03:28:10.412750 13086 raft_consensus.cc:374] T 00000000000000000000000000000000 P f0c134cae7974fed9683afccfc749107 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:28:10.412868 13086 raft_consensus.cc:729] T 00000000000000000000000000000000 P f0c134cae7974fed9683afccfc749107 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0c134cae7974fed9683afccfc749107, State: Initialized, Role: FOLLOWER
I20260810 03:28:10.413452 13086 consensus_queue.cc:260] T 00000000000000000000000000000000 P f0c134cae7974fed9683afccfc749107 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0c134cae7974fed9683afccfc749107" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 32815 } }
I20260810 03:28:10.413763 13086 raft_consensus.cc:388] T 00000000000000000000000000000000 P f0c134cae7974fed9683afccfc749107 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 03:28:10.413902 13086 raft_consensus.cc:482] T 00000000000000000000000000000000 P f0c134cae7974fed9683afccfc749107 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 03:28:10.414130 13086 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f0c134cae7974fed9683afccfc749107 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:28:10.416714 13086 raft_consensus.cc:504] T 00000000000000000000000000000000 P f0c134cae7974fed9683afccfc749107 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0c134cae7974fed9683afccfc749107" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 32815 } }
I20260810 03:28:10.417161 13086 leader_election.cc:302] T 00000000000000000000000000000000 P f0c134cae7974fed9683afccfc749107 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f0c134cae7974fed9683afccfc749107; no voters:
I20260810 03:28:10.417785 13086 leader_election.cc:288] T 00000000000000000000000000000000 P f0c134cae7974fed9683afccfc749107 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 03:28:10.417991 13092 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f0c134cae7974fed9683afccfc749107 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:28:10.419175 13092 raft_consensus.cc:686] T 00000000000000000000000000000000 P f0c134cae7974fed9683afccfc749107 [term 1 LEADER]: Becoming Leader. State: Replica: f0c134cae7974fed9683afccfc749107, State: Running, Role: LEADER
I20260810 03:28:10.419723 13092 consensus_queue.cc:237] T 00000000000000000000000000000000 P f0c134cae7974fed9683afccfc749107 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0c134cae7974fed9683afccfc749107" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 32815 } }
I20260810 03:28:10.420078 13086 sys_catalog.cc:564] T 00000000000000000000000000000000 P f0c134cae7974fed9683afccfc749107 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:28:10.422600 13096 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0c134cae7974fed9683afccfc749107 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f0c134cae7974fed9683afccfc749107. Latest consensus state: current_term: 1 leader_uuid: "f0c134cae7974fed9683afccfc749107" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0c134cae7974fed9683afccfc749107" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 32815 } } }
I20260810 03:28:10.422966 13096 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0c134cae7974fed9683afccfc749107 [sys.catalog]: This master's current role is: LEADER
I20260810 03:28:10.423399 13094 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0c134cae7974fed9683afccfc749107 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f0c134cae7974fed9683afccfc749107" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0c134cae7974fed9683afccfc749107" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 32815 } } }
I20260810 03:28:10.423728 13094 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0c134cae7974fed9683afccfc749107 [sys.catalog]: This master's current role is: LEADER
I20260810 03:28:10.423995 13106 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 03:28:10.431150 13106 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 03:28:10.442595 13086 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786332336319544-28292-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260810 03:28:10.443500 13106 catalog_manager.cc:1347] Generated new cluster ID: 89b8295a4ee442f6beee63e1fb25bbb7
I20260810 03:28:10.443601 13106 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 03:28:10.466473 13106 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 03:28:10.467967 13106 catalog_manager.cc:1504] Loading token signing keys...
I20260810 03:28:10.477015 13106 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f0c134cae7974fed9683afccfc749107: Generated new TSK 0
I20260810 03:28:10.478032 13106 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.161.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.161.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.161.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.1.keytab.
Entry for principal kudu/127.27.161.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.1.keytab.
I20260810 03:28:11.335476 28292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iBH5/build/asan/bin/kudu
/tmp/dist-test-task24iBH5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786332336319544-28292-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786332336319544-28292-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786332336319544-28292-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786332336319544-28292-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.161.20:55767/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.161.1:0
--local_ip_for_outbound_sockets=127.27.161.1
--webserver_interface=127.27.161.1
--webserver_port=0
--tserver_master_addrs=127.27.161.62:32815
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.1.keytab
--principal=kudu/127.27.161.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.161.20:43399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786332336319544-28292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786332336319544-28292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786332336319544-28292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 03:28:11 dist-test-slave-vpvm krb5kdc[12295](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786332491, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.161.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260810 03:28:11.869940 13155 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:28:11.870613 13155 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:28:11.870823 13155 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:28:11.883759 13155 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:28:11.883953 13155 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:28:11.884302 13155 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.161.1
I20260810 03:28:11.902220 13155 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.161.20:43399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786332336319544-28292-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786332336319544-28292-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.27.161.20:55767/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.1.keytab
--principal=kudu/127.27.161.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.161.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786332336319544-28292-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.161.1
--webserver_port=0
--tserver_master_addrs=127.27.161.62:32815
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.161.1
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786332336319544-28292-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:28:11.903733 13155 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:28:11.905360 13155 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:28:11.930421 13175 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:28:11.931694 13172 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:28:11.933717 13177 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:28:11.934180 13155 server_base.cc:1034] running on GCE node
I20260810 03:28:11.934990 13155 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:28:11.936100 13155 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:28:11.937376 13155 hybrid_clock.cc:648] HybridClock initialized: now 1786332491937300 us; error 66 us; skew 500 ppm
I20260810 03:28:11.937755 13155 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:28:11 dist-test-slave-vpvm krb5kdc[12295](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332491, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.161.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:28:11.942000 13155 init.cc:377] Logged in from keytab as kudu/127.27.161.1@KRBTEST.COM (short username kudu)
I20260810 03:28:11.944561 13155 webserver.cc:460] Webserver started at http://127.27.161.1:42029/ using document root <none> and password file <none>
I20260810 03:28:11.945716 13155 fs_manager.cc:359] Metadata directory not provided
I20260810 03:28:11.945905 13155 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:28:11.946381 13155 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 03:28:11 dist-test-slave-vpvm krb5kdc[12295](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332491, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.161.1@KRBTEST.COM for HTTP/127.27.161.20@KRBTEST.COM
I20260810 03:28:11.969942 13155 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786332336319544-28292-0/minicluster-data/ts-0/data/instance:
uuid: "89958a9959614a45b8bcb836677733f7"
format_stamp: "Formatted at 2026-08-10 03:28:11 on dist-test-slave-vpvm"
server_key: "44f10c2761fcb258c7943e8b42d522b5"
server_key_iv: "fdc084948ea4992e850ff3672c499bc1"
server_key_version: "kuduclusterkey@0"
I20260810 03:28:11.971752 13155 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786332336319544-28292-0/minicluster-data/ts-0/wal/instance:
uuid: "89958a9959614a45b8bcb836677733f7"
format_stamp: "Formatted at 2026-08-10 03:28:11 on dist-test-slave-vpvm"
server_key: "44f10c2761fcb258c7943e8b42d522b5"
server_key_iv: "fdc084948ea4992e850ff3672c499bc1"
server_key_version: "kuduclusterkey@0"
I20260810 03:28:11.982676 13155 fs_manager.cc:691] Time spent creating directory manager: real 0.010s user 0.010s sys 0.001s
I20260810 03:28:12.017323 13186 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:28:12.019711 13155 fs_manager.cc:725] Time spent opening block manager: real 0.010s user 0.003s sys 0.006s
I20260810 03:28:12.020069 13155 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786332336319544-28292-0/minicluster-data/ts-0/data,/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786332336319544-28292-0/minicluster-data/ts-0/wal
uuid: "89958a9959614a45b8bcb836677733f7"
format_stamp: "Formatted at 2026-08-10 03:28:11 on dist-test-slave-vpvm"
server_key: "44f10c2761fcb258c7943e8b42d522b5"
server_key_iv: "fdc084948ea4992e850ff3672c499bc1"
server_key_version: "kuduclusterkey@0"
I20260810 03:28:12.020498 13155 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786332336319544-28292-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786332336319544-28292-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786332336319544-28292-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:28:12.075181 13155 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:28:12.083083 13155 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:28:12.083662 13155 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:28:12.085284 13155 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:28:12.091279 13155 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:28:12.091490 13155 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260810 03:28:12.091818 13155 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:28:12.091982 13155 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 03:28:12 dist-test-slave-vpvm krb5kdc[12295](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332491, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.161.62@KRBTEST.COM for HTTP/127.27.161.20@KRBTEST.COM
I20260810 03:28:12.497895 13155 rpc_server.cc:307] RPC server started. Bound to: 127.27.161.1:44119
I20260810 03:28:12.500057 13358 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.161.1:44119 every 8 connection(s)
I20260810 03:28:12.500250 13155 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786332336319544-28292-0/minicluster-data/ts-0/data/info.pb
I20260810 03:28:12.501582 28292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iBH5/build/asan/bin/kudu as pid 13155
I20260810 03:28:12.501806 28292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786332336319544-28292-0/minicluster-data/ts-0/wal/instance
I20260810 03:28:12.502679 28292 external_mini_cluster.cc:1428] Setting key 6edb260d4bd69872edbe14a168ff089f
Aug 10 03:28:12 dist-test-slave-vpvm krb5kdc[12295](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332491, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.161.1@KRBTEST.COM for kudu/127.27.161.62@KRBTEST.COM
I20260810 03:28:12.552634 13360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:28:12.510267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.161.1:60917 (local address 127.27.161.62:32815)
0810 03:28:12.521794 (+ 11527us) server_negotiation.cc:206] Beginning negotiation
0810 03:28:12.521802 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 03:28:12.526256 (+ 4454us) server_negotiation.cc:407] Connection header received
0810 03:28:12.529584 (+ 3328us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:28:12.529591 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:28:12.529712 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:28:12.529825 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:28:12.533076 (+ 3251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:28:12.534447 (+ 1371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:28:12.537093 (+ 2646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:28:12.537722 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:28:12.537829 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:28:12.543514 (+ 5685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:28:12.543523 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:28:12.543528 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:28:12.543580 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:28:12.547790 (+ 4210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:28:12.549041 (+ 1251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:28:12.549048 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:28:12.549052 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:28:12.549261 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:28:12.550140 (+ 879us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:28:12.550146 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:28:12.550149 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:28:12.550633 (+ 484us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:28:12.550766 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0810 03:28:12.551891 (+ 1125us) server_negotiation.cc:299] Negotiation successful
0810 03:28:12.552170 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11342,"thread_start_us":11203,"threads_started":1}
I20260810 03:28:12.555478 13362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:28:12.510936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:32815 (local address 127.27.161.1:60917)
0810 03:28:12.526022 (+ 15086us) negotiation.cc:107] Waiting for socket to connect
0810 03:28:12.526100 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0810 03:28:12.529103 (+ 3003us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:28:12.530894 (+ 1791us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:28:12.530908 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:28:12.531669 (+ 761us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:28:12.532809 (+ 1140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:28:12.532842 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:28:12.534712 (+ 1870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:28:12.534718 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:28:12.536886 (+ 2168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:28:12.536907 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:28:12.538277 (+ 1370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:28:12.538285 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:28:12.538595 (+ 310us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:28:12.539524 (+ 929us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:28:12.539556 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:28:12.543066 (+ 3510us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:28:12.548078 (+ 5012us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:28:12.548090 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:28:12.548112 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:28:12.548862 (+ 750us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:28:12.549455 (+ 593us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:28:12.549460 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:28:12.549464 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:28:12.549984 (+ 520us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:28:12.550894 (+ 910us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:28:12.550903 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:28:12.551407 (+ 504us) client_negotiation.cc:769] Sending connection context
0810 03:28:12.553247 (+ 1840us) client_negotiation.cc:240] Negotiation successful
0810 03:28:12.553868 (+ 621us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14664,"thread_start_us":18563,"threads_started":1}
I20260810 03:28:12.558277 13359 heartbeater.cc:347] Connected to a master server at 127.27.161.62:32815
I20260810 03:28:12.558753 13359 heartbeater.cc:464] Registering TS with master...
I20260810 03:28:12.560217 13359 heartbeater.cc:511] Master 127.27.161.62:32815 requested a full tablet report, sending...
I20260810 03:28:12.564239 13030 ts_manager.cc:194] Registered new tserver with Master: 89958a9959614a45b8bcb836677733f7 (127.27.161.1:44119)
I20260810 03:28:12.566620 13030 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.161.1@KRBTEST.COM'} at 127.27.161.1:60917
WARNING: no policy specified for kudu/127.27.161.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.161.2@KRBTEST.COM" created.
Aug 10 03:28:12 dist-test-slave-vpvm krb5kdc[12295](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332491, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.161.62@KRBTEST.COM for HTTP/127.27.161.20@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.161.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.2.keytab.
Entry for principal kudu/127.27.161.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.2.keytab.
W20260810 03:28:12.630950 28292 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.27.161.2: real 0.125s user 0.005s sys 0.024s
I20260810 03:28:12.632121 28292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iBH5/build/asan/bin/kudu
/tmp/dist-test-task24iBH5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786332336319544-28292-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786332336319544-28292-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786332336319544-28292-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786332336319544-28292-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.161.20:55767/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.161.2:0
--local_ip_for_outbound_sockets=127.27.161.2
--webserver_interface=127.27.161.2
--webserver_port=0
--tserver_master_addrs=127.27.161.62:32815
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.2.keytab
--principal=kudu/127.27.161.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.161.20:43399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786332336319544-28292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786332336319544-28292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786332336319544-28292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:28:13.135946 13373 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:28:13.136270 13373 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:28:13.136412 13373 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:28:13.145435 13373 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:28:13.145581 13373 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:28:13.145743 13373 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.161.2
I20260810 03:28:13.157492 13373 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.161.20:43399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786332336319544-28292-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786332336319544-28292-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.27.161.20:55767/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.2.keytab
--principal=kudu/127.27.161.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.161.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786332336319544-28292-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.161.2
--webserver_port=0
--tserver_master_addrs=127.27.161.62:32815
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.161.2
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786332336319544-28292-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:28:13.159044 13373 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:28:13.160419 13373 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:28:13.176509 13400 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:28:13.176465 13402 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:28:13.176445 13408 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:28:13.177707 13373 server_base.cc:1034] running on GCE node
I20260810 03:28:13.178444 13373 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:28:13.179538 13373 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:28:13.180794 13373 hybrid_clock.cc:648] HybridClock initialized: now 1786332493180737 us; error 44 us; skew 500 ppm
I20260810 03:28:13.181226 13373 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:28:13 dist-test-slave-vpvm krb5kdc[12295](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332493, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.161.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:28:13.184965 13373 init.cc:377] Logged in from keytab as kudu/127.27.161.2@KRBTEST.COM (short username kudu)
I20260810 03:28:13.186733 13373 webserver.cc:460] Webserver started at http://127.27.161.2:33831/ using document root <none> and password file <none>
I20260810 03:28:13.187481 13373 fs_manager.cc:359] Metadata directory not provided
I20260810 03:28:13.187625 13373 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:28:13.187944 13373 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 03:28:13 dist-test-slave-vpvm krb5kdc[12295](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332493, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.161.2@KRBTEST.COM for HTTP/127.27.161.20@KRBTEST.COM
I20260810 03:28:13.208015 13373 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786332336319544-28292-0/minicluster-data/ts-1/data/instance:
uuid: "6420226252c2434394d828b33da15322"
format_stamp: "Formatted at 2026-08-10 03:28:13 on dist-test-slave-vpvm"
server_key: "a8cef2247e6ef53bf0b9b41a54080d23"
server_key_iv: "128838d0a9bf69df082e9293d5a1fe2e"
server_key_version: "kuduclusterkey@0"
I20260810 03:28:13.209513 13373 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786332336319544-28292-0/minicluster-data/ts-1/wal/instance:
uuid: "6420226252c2434394d828b33da15322"
format_stamp: "Formatted at 2026-08-10 03:28:13 on dist-test-slave-vpvm"
server_key: "a8cef2247e6ef53bf0b9b41a54080d23"
server_key_iv: "128838d0a9bf69df082e9293d5a1fe2e"
server_key_version: "kuduclusterkey@0"
I20260810 03:28:13.218055 13373 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.006s sys 0.001s
I20260810 03:28:13.237920 13419 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:28:13.239928 13373 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260810 03:28:13.240162 13373 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786332336319544-28292-0/minicluster-data/ts-1/data,/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786332336319544-28292-0/minicluster-data/ts-1/wal
uuid: "6420226252c2434394d828b33da15322"
format_stamp: "Formatted at 2026-08-10 03:28:13 on dist-test-slave-vpvm"
server_key: "a8cef2247e6ef53bf0b9b41a54080d23"
server_key_iv: "128838d0a9bf69df082e9293d5a1fe2e"
server_key_version: "kuduclusterkey@0"
I20260810 03:28:13.240480 13373 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786332336319544-28292-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786332336319544-28292-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786332336319544-28292-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:28:13.291343 13373 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:28:13.296164 13373 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:28:13.296556 13373 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:28:13.297722 13373 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:28:13.299602 13373 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:28:13.299695 13373 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:28:13.299917 13373 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:28:13.300021 13373 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:28:13.358433 13373 rpc_server.cc:307] RPC server started. Bound to: 127.27.161.2:33371
I20260810 03:28:13.358454 13593 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.161.2:33371 every 8 connection(s)
I20260810 03:28:13.359998 13373 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786332336319544-28292-0/minicluster-data/ts-1/data/info.pb
I20260810 03:28:13.368716 28292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iBH5/build/asan/bin/kudu as pid 13373
I20260810 03:28:13.368897 28292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786332336319544-28292-0/minicluster-data/ts-1/wal/instance
I20260810 03:28:13.369436 28292 external_mini_cluster.cc:1428] Setting key 82e4d80e5444df11da939e307e222709
Aug 10 03:28:13 dist-test-slave-vpvm krb5kdc[12295](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332493, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.161.2@KRBTEST.COM for kudu/127.27.161.62@KRBTEST.COM
I20260810 03:28:13.386448 13597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:28:13.363481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.161.2:50471 (local address 127.27.161.62:32815)
0810 03:28:13.364156 (+ 675us) server_negotiation.cc:206] Beginning negotiation
0810 03:28:13.364161 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:28:13.365538 (+ 1377us) server_negotiation.cc:407] Connection header received
0810 03:28:13.367806 (+ 2268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:28:13.367813 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:28:13.367904 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:28:13.368004 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:28:13.370182 (+ 2178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:28:13.371615 (+ 1433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:28:13.374017 (+ 2402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:28:13.374573 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:28:13.374691 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:28:13.379318 (+ 4627us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:28:13.379324 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:28:13.379327 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:28:13.379367 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:28:13.382617 (+ 3250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:28:13.383583 (+ 966us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:28:13.383588 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:28:13.383590 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:28:13.383819 (+ 229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:28:13.384364 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:28:13.384368 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:28:13.384371 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:28:13.384753 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:28:13.384916 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0810 03:28:13.385725 (+ 809us) server_negotiation.cc:299] Negotiation successful
0810 03:28:13.386026 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":544,"thread_start_us":423,"threads_started":1}
I20260810 03:28:13.387944 13600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:28:13.364402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:32815 (local address 127.27.161.2:50471)
0810 03:28:13.365344 (+ 942us) negotiation.cc:107] Waiting for socket to connect
0810 03:28:13.365388 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0810 03:28:13.367483 (+ 2095us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:28:13.368315 (+ 832us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:28:13.368325 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:28:13.368972 (+ 647us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:28:13.369945 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:28:13.370007 (+ 62us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:28:13.371819 (+ 1812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:28:13.371825 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:28:13.373791 (+ 1966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:28:13.373821 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:28:13.374826 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:28:13.374833 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:28:13.375132 (+ 299us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:28:13.375897 (+ 765us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:28:13.375928 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:28:13.379044 (+ 3116us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:28:13.382808 (+ 3764us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:28:13.382820 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:28:13.382837 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:28:13.383435 (+ 598us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:28:13.383947 (+ 512us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:28:13.383953 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:28:13.383956 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:28:13.384239 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:28:13.384903 (+ 664us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:28:13.384913 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:28:13.385309 (+ 396us) client_negotiation.cc:769] Sending connection context
0810 03:28:13.385575 (+ 266us) client_negotiation.cc:240] Negotiation successful
0810 03:28:13.386205 (+ 630us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":655,"thread_start_us":403,"threads_started":1}
I20260810 03:28:13.389787 13594 heartbeater.cc:347] Connected to a master server at 127.27.161.62:32815
I20260810 03:28:13.390161 13594 heartbeater.cc:464] Registering TS with master...
I20260810 03:28:13.391114 13594 heartbeater.cc:511] Master 127.27.161.62:32815 requested a full tablet report, sending...
I20260810 03:28:13.392933 13030 ts_manager.cc:194] Registered new tserver with Master: 6420226252c2434394d828b33da15322 (127.27.161.2:33371)
I20260810 03:28:13.394098 13030 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.161.2@KRBTEST.COM'} at 127.27.161.2:50471
WARNING: no policy specified for kudu/127.27.161.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.161.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.161.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.3.keytab.
Entry for principal kudu/127.27.161.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.3.keytab.
I20260810 03:28:13.441697 28292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iBH5/build/asan/bin/kudu
/tmp/dist-test-task24iBH5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786332336319544-28292-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786332336319544-28292-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786332336319544-28292-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786332336319544-28292-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.161.20:55767/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.161.3:0
--local_ip_for_outbound_sockets=127.27.161.3
--webserver_interface=127.27.161.3
--webserver_port=0
--tserver_master_addrs=127.27.161.62:32815
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.3.keytab
--principal=kudu/127.27.161.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.161.20:43399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786332336319544-28292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786332336319544-28292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786332336319544-28292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 03:28:13.571327 13359 heartbeater.cc:503] Master 127.27.161.62:32815 was elected leader, sending a full tablet report...
W20260810 03:28:13.853817 13603 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:28:13.854456 13603 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:28:13.854636 13603 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:28:13.862778 13603 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:28:13.862922 13603 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:28:13.863093 13603 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.161.3
I20260810 03:28:13.874477 13603 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.161.20:43399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786332336319544-28292-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786332336319544-28292-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.27.161.20:55767/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.3.keytab
--principal=kudu/127.27.161.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.161.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786332336319544-28292-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.161.3
--webserver_port=0
--tserver_master_addrs=127.27.161.62:32815
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.161.3
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786332336319544-28292-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:28:13.875948 13603 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:28:13.877323 13603 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:28:13.893082 13613 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:28:13.893222 13618 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:28:13.893347 13621 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:28:13.894872 13603 server_base.cc:1034] running on GCE node
I20260810 03:28:13.895640 13603 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:28:13.896778 13603 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:28:13.898075 13603 hybrid_clock.cc:648] HybridClock initialized: now 1786332493898020 us; error 41 us; skew 500 ppm
I20260810 03:28:13.898455 13603 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:28:13 dist-test-slave-vpvm krb5kdc[12295](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332493, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.161.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:28:13.902202 13603 init.cc:377] Logged in from keytab as kudu/127.27.161.3@KRBTEST.COM (short username kudu)
I20260810 03:28:13.903995 13603 webserver.cc:460] Webserver started at http://127.27.161.3:42153/ using document root <none> and password file <none>
I20260810 03:28:13.904832 13603 fs_manager.cc:359] Metadata directory not provided
I20260810 03:28:13.904968 13603 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:28:13.905300 13603 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 03:28:13 dist-test-slave-vpvm krb5kdc[12295](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332493, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.161.3@KRBTEST.COM for HTTP/127.27.161.20@KRBTEST.COM
I20260810 03:28:13.920424 13603 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786332336319544-28292-0/minicluster-data/ts-2/data/instance:
uuid: "30879b54184e41e184a8fa62a3bad0a4"
format_stamp: "Formatted at 2026-08-10 03:28:13 on dist-test-slave-vpvm"
server_key: "8e258711ec4d59b50b9399acf34c0329"
server_key_iv: "4aae53ee18377242d850b15667363d78"
server_key_version: "kuduclusterkey@0"
I20260810 03:28:13.921514 13603 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786332336319544-28292-0/minicluster-data/ts-2/wal/instance:
uuid: "30879b54184e41e184a8fa62a3bad0a4"
format_stamp: "Formatted at 2026-08-10 03:28:13 on dist-test-slave-vpvm"
server_key: "8e258711ec4d59b50b9399acf34c0329"
server_key_iv: "4aae53ee18377242d850b15667363d78"
server_key_version: "kuduclusterkey@0"
I20260810 03:28:13.928099 13603 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.000s sys 0.004s
I20260810 03:28:13.945766 13633 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:28:13.948016 13603 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.002s
I20260810 03:28:13.948268 13603 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786332336319544-28292-0/minicluster-data/ts-2/data,/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786332336319544-28292-0/minicluster-data/ts-2/wal
uuid: "30879b54184e41e184a8fa62a3bad0a4"
format_stamp: "Formatted at 2026-08-10 03:28:13 on dist-test-slave-vpvm"
server_key: "8e258711ec4d59b50b9399acf34c0329"
server_key_iv: "4aae53ee18377242d850b15667363d78"
server_key_version: "kuduclusterkey@0"
I20260810 03:28:13.948796 13603 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786332336319544-28292-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786332336319544-28292-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786332336319544-28292-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:28:13.982177 13603 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:28:13.987628 13603 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:28:13.987931 13603 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:28:13.988950 13603 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:28:13.990743 13603 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:28:13.990823 13603 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:28:13.990922 13603 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:28:13.990971 13603 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:28:14.041852 13603 rpc_server.cc:307] RPC server started. Bound to: 127.27.161.3:41829
I20260810 03:28:14.041865 13809 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.161.3:41829 every 8 connection(s)
I20260810 03:28:14.043467 13603 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786332336319544-28292-0/minicluster-data/ts-2/data/info.pb
I20260810 03:28:14.046864 28292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iBH5/build/asan/bin/kudu as pid 13603
I20260810 03:28:14.047053 28292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786332336319544-28292-0/minicluster-data/ts-2/wal/instance
I20260810 03:28:14.047626 28292 external_mini_cluster.cc:1428] Setting key a40fad3bc667739f21b9b386d9662903
Aug 10 03:28:14 dist-test-slave-vpvm krb5kdc[12295](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332493, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.161.3@KRBTEST.COM for kudu/127.27.161.62@KRBTEST.COM
I20260810 03:28:14.069769 13815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:28:14.046455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.161.3:52669 (local address 127.27.161.62:32815)
0810 03:28:14.047132 (+ 677us) server_negotiation.cc:206] Beginning negotiation
0810 03:28:14.047139 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:28:14.049026 (+ 1887us) server_negotiation.cc:407] Connection header received
0810 03:28:14.051601 (+ 2575us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:28:14.051607 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:28:14.051703 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:28:14.051799 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:28:14.053661 (+ 1862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:28:14.054582 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:28:14.056460 (+ 1878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:28:14.057350 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:28:14.057477 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:28:14.062264 (+ 4787us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:28:14.062271 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:28:14.062275 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:28:14.062317 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:28:14.066086 (+ 3769us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:28:14.067196 (+ 1110us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:28:14.067201 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:28:14.067203 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:28:14.067347 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:28:14.067884 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:28:14.067888 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:28:14.067891 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:28:14.068213 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:28:14.068360 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0810 03:28:14.069135 (+ 775us) server_negotiation.cc:299] Negotiation successful
0810 03:28:14.069401 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":543,"thread_start_us":390,"threads_started":1}
I20260810 03:28:14.070837 13816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:28:14.047425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:32815 (local address 127.27.161.3:52669)
0810 03:28:14.048841 (+ 1416us) negotiation.cc:107] Waiting for socket to connect
0810 03:28:14.048897 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0810 03:28:14.051254 (+ 2357us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:28:14.052074 (+ 820us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:28:14.052083 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:28:14.052557 (+ 474us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:28:14.053471 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:28:14.053495 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:28:14.054782 (+ 1287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:28:14.054786 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:28:14.056235 (+ 1449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:28:14.056259 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:28:14.057696 (+ 1437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:28:14.057708 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:28:14.058171 (+ 463us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:28:14.059070 (+ 899us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:28:14.059101 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:28:14.061857 (+ 2756us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:28:14.066275 (+ 4418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:28:14.066285 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:28:14.066302 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:28:14.067050 (+ 748us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:28:14.067467 (+ 417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:28:14.067472 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:28:14.067475 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:28:14.067761 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:28:14.068372 (+ 611us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:28:14.068382 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:28:14.068751 (+ 369us) client_negotiation.cc:769] Sending connection context
0810 03:28:14.069022 (+ 271us) client_negotiation.cc:240] Negotiation successful
0810 03:28:14.069597 (+ 575us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1075,"thread_start_us":757,"threads_started":1}
I20260810 03:28:14.072402 13811 heartbeater.cc:347] Connected to a master server at 127.27.161.62:32815
I20260810 03:28:14.072738 13811 heartbeater.cc:464] Registering TS with master...
I20260810 03:28:14.073530 13811 heartbeater.cc:511] Master 127.27.161.62:32815 requested a full tablet report, sending...
I20260810 03:28:14.075784 13030 ts_manager.cc:194] Registered new tserver with Master: 30879b54184e41e184a8fa62a3bad0a4 (127.27.161.3:41829)
I20260810 03:28:14.076902 13030 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.161.3@KRBTEST.COM'} at 127.27.161.3:52669
I20260810 03:28:14.077497 28292 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 03:28:14.097561 13815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:28:14.083067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58116 (local address 127.27.161.62:32815)
0810 03:28:14.083262 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0810 03:28:14.083270 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 03:28:14.083774 (+ 504us) server_negotiation.cc:407] Connection header received
0810 03:28:14.083951 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:28:14.083956 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:28:14.084037 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:28:14.084133 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:28:14.085794 (+ 1661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:28:14.086648 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:28:14.088090 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:28:14.088694 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:28:14.088815 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:28:14.091593 (+ 2778us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:28:14.091599 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:28:14.091603 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:28:14.091641 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:28:14.094567 (+ 2926us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:28:14.095257 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:28:14.095261 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:28:14.095263 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:28:14.095439 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:28:14.095959 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:28:14.095963 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:28:14.095965 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:28:14.096312 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:28:14.096560 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0810 03:28:14.096905 (+ 345us) server_negotiation.cc:299] Negotiation successful
0810 03:28:14.097158 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260810 03:28:14.101652 13030 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58116:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 03:28:14.110049 13030 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 03:28:14.151171 13838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:28:14.124867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42604 (local address 127.27.161.3:41829)
0810 03:28:14.138240 (+ 13373us) server_negotiation.cc:206] Beginning negotiation
0810 03:28:14.138253 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 03:28:14.138301 (+ 48us) server_negotiation.cc:407] Connection header received
0810 03:28:14.138431 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:28:14.138441 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:28:14.138779 (+ 338us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:28:14.140677 (+ 1898us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:28:14.141214 (+ 537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:28:14.143234 (+ 2020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:28:14.146070 (+ 2836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:28:14.147768 (+ 1698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:28:14.148202 (+ 434us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:28:14.148477 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0810 03:28:14.150561 (+ 2084us) server_negotiation.cc:299] Negotiation successful
0810 03:28:14.150754 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13105,"thread_start_us":14103,"threads_started":1}
I20260810 03:28:14.151275 13835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:28:14.124399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.3:41829 (local address 127.0.0.1:42604)
0810 03:28:14.125459 (+ 1060us) negotiation.cc:107] Waiting for socket to connect
0810 03:28:14.125509 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0810 03:28:14.125703 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:28:14.139353 (+ 13650us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:28:14.139363 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:28:14.139417 (+ 54us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:28:14.140434 (+ 1017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:28:14.140450 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:28:14.143418 (+ 2968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:28:14.143424 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:28:14.145893 (+ 2469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:28:14.145911 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:28:14.148034 (+ 2123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:28:14.148041 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:28:14.150270 (+ 2229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:28:14.150328 (+ 58us) client_negotiation.cc:769] Sending connection context
0810 03:28:14.150493 (+ 165us) client_negotiation.cc:240] Negotiation successful
0810 03:28:14.150742 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":797,"mutex_wait_us":54,"thread_start_us":16184,"threads_started":1}
I20260810 03:28:14.151871 13842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:28:14.126381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.2:33371 (local address 127.0.0.1:54526)
0810 03:28:14.142845 (+ 16464us) negotiation.cc:107] Waiting for socket to connect
0810 03:28:14.142863 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 03:28:14.142985 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:28:14.143702 (+ 717us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:28:14.143707 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:28:14.143731 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:28:14.144331 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:28:14.144342 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:28:14.146249 (+ 1907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:28:14.146254 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:28:14.148492 (+ 2238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:28:14.148510 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:28:14.151061 (+ 2551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:28:14.151067 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:28:14.151279 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:28:14.151316 (+ 37us) client_negotiation.cc:769] Sending connection context
0810 03:28:14.151402 (+ 86us) client_negotiation.cc:240] Negotiation successful
0810 03:28:14.151536 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16342,"thread_start_us":16158,"threads_started":1}
I20260810 03:28:14.152964 13843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:28:14.126685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54526 (local address 127.27.161.2:33371)
0810 03:28:14.141564 (+ 14879us) server_negotiation.cc:206] Beginning negotiation
0810 03:28:14.141573 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 03:28:14.142995 (+ 1422us) server_negotiation.cc:407] Connection header received
0810 03:28:14.143132 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:28:14.143142 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:28:14.143452 (+ 310us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:28:14.143700 (+ 248us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:28:14.144623 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:28:14.146099 (+ 1476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:28:14.148750 (+ 2651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:28:14.150551 (+ 1801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:28:14.150681 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:28:14.150895 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0810 03:28:14.152268 (+ 1373us) server_negotiation.cc:299] Negotiation successful
0810 03:28:14.152464 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14688,"thread_start_us":15092,"threads_started":1}
I20260810 03:28:14.153198 13831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:28:14.124722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.1:44119 (local address 127.0.0.1:44374)
0810 03:28:14.140297 (+ 15575us) negotiation.cc:107] Waiting for socket to connect
0810 03:28:14.140347 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0810 03:28:14.140515 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:28:14.144267 (+ 3752us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:28:14.144273 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:28:14.144303 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:28:14.145151 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:28:14.145166 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:28:14.147758 (+ 2592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:28:14.147763 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:28:14.150412 (+ 2649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:28:14.150429 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:28:14.152509 (+ 2080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:28:14.152515 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:28:14.152694 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:28:14.152726 (+ 32us) client_negotiation.cc:769] Sending connection context
0810 03:28:14.152802 (+ 76us) client_negotiation.cc:240] Negotiation successful
0810 03:28:14.152897 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15367,"mutex_wait_us":56,"thread_start_us":912,"threads_started":1}
I20260810 03:28:14.154933 13839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:28:14.125432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44374 (local address 127.27.161.1:44119)
0810 03:28:14.143668 (+ 18236us) server_negotiation.cc:206] Beginning negotiation
0810 03:28:14.143689 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0810 03:28:14.143720 (+ 31us) server_negotiation.cc:407] Connection header received
0810 03:28:14.143825 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:28:14.143832 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:28:14.144049 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:28:14.144249 (+ 200us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:28:14.145348 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:28:14.147576 (+ 2228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:28:14.150628 (+ 3052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:28:14.152268 (+ 1640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:28:14.152422 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:28:14.154160 (+ 1738us) server_negotiation.cc:1035] Waiting for connection context
0810 03:28:14.154252 (+ 92us) server_negotiation.cc:299] Negotiation successful
0810 03:28:14.154433 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18001,"thread_start_us":17787,"threads_started":1}
I20260810 03:28:14.155985 13718 tablet_service.cc:1467] Processing CreateTablet for tablet 44e2930c1db3427e94f6d5b361ff8cb8 (DEFAULT_TABLE table=test-table [id=1bc586ac3f7c45a59f240a1d98f2db9c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:28:14.156387 13491 tablet_service.cc:1467] Processing CreateTablet for tablet 44e2930c1db3427e94f6d5b361ff8cb8 (DEFAULT_TABLE table=test-table [id=1bc586ac3f7c45a59f240a1d98f2db9c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:28:14.157469 13718 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44e2930c1db3427e94f6d5b361ff8cb8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:28:14.157585 13491 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44e2930c1db3427e94f6d5b361ff8cb8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:28:14.165813 13259 tablet_service.cc:1467] Processing CreateTablet for tablet 44e2930c1db3427e94f6d5b361ff8cb8 (DEFAULT_TABLE table=test-table [id=1bc586ac3f7c45a59f240a1d98f2db9c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:28:14.167300 13259 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44e2930c1db3427e94f6d5b361ff8cb8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:28:14.173481 13845 tablet_bootstrap.cc:492] T 44e2930c1db3427e94f6d5b361ff8cb8 P 6420226252c2434394d828b33da15322: Bootstrap starting.
I20260810 03:28:14.174468 13847 tablet_bootstrap.cc:492] T 44e2930c1db3427e94f6d5b361ff8cb8 P 30879b54184e41e184a8fa62a3bad0a4: Bootstrap starting.
I20260810 03:28:14.178622 13845 tablet_bootstrap.cc:654] T 44e2930c1db3427e94f6d5b361ff8cb8 P 6420226252c2434394d828b33da15322: Neither blocks nor log segments found. Creating new log.
I20260810 03:28:14.178617 13847 tablet_bootstrap.cc:654] T 44e2930c1db3427e94f6d5b361ff8cb8 P 30879b54184e41e184a8fa62a3bad0a4: Neither blocks nor log segments found. Creating new log.
I20260810 03:28:14.179949 13845 log.cc:824] T 44e2930c1db3427e94f6d5b361ff8cb8 P 6420226252c2434394d828b33da15322: Log is configured to *not* fsync() on all Append() calls
I20260810 03:28:14.180121 13847 log.cc:824] T 44e2930c1db3427e94f6d5b361ff8cb8 P 30879b54184e41e184a8fa62a3bad0a4: Log is configured to *not* fsync() on all Append() calls
I20260810 03:28:14.183969 13845 tablet_bootstrap.cc:492] T 44e2930c1db3427e94f6d5b361ff8cb8 P 6420226252c2434394d828b33da15322: No bootstrap required, opened a new log
I20260810 03:28:14.184139 13847 tablet_bootstrap.cc:492] T 44e2930c1db3427e94f6d5b361ff8cb8 P 30879b54184e41e184a8fa62a3bad0a4: No bootstrap required, opened a new log
I20260810 03:28:14.184289 13845 ts_tablet_manager.cc:1397] T 44e2930c1db3427e94f6d5b361ff8cb8 P 6420226252c2434394d828b33da15322: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20260810 03:28:14.184417 13847 ts_tablet_manager.cc:1397] T 44e2930c1db3427e94f6d5b361ff8cb8 P 30879b54184e41e184a8fa62a3bad0a4: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.003s
I20260810 03:28:14.184645 13850 tablet_bootstrap.cc:492] T 44e2930c1db3427e94f6d5b361ff8cb8 P 89958a9959614a45b8bcb836677733f7: Bootstrap starting.
I20260810 03:28:14.190795 13850 tablet_bootstrap.cc:654] T 44e2930c1db3427e94f6d5b361ff8cb8 P 89958a9959614a45b8bcb836677733f7: Neither blocks nor log segments found. Creating new log.
I20260810 03:28:14.192728 13850 log.cc:824] T 44e2930c1db3427e94f6d5b361ff8cb8 P 89958a9959614a45b8bcb836677733f7: Log is configured to *not* fsync() on all Append() calls
I20260810 03:28:14.194331 13845 raft_consensus.cc:348] T 44e2930c1db3427e94f6d5b361ff8cb8 P 6420226252c2434394d828b33da15322 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30879b54184e41e184a8fa62a3bad0a4" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 41829 } } peers { permanent_uuid: "89958a9959614a45b8bcb836677733f7" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 44119 } } peers { permanent_uuid: "6420226252c2434394d828b33da15322" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 33371 } }
I20260810 03:28:14.194869 13845 raft_consensus.cc:374] T 44e2930c1db3427e94f6d5b361ff8cb8 P 6420226252c2434394d828b33da15322 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:28:14.195020 13845 raft_consensus.cc:729] T 44e2930c1db3427e94f6d5b361ff8cb8 P 6420226252c2434394d828b33da15322 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6420226252c2434394d828b33da15322, State: Initialized, Role: FOLLOWER
I20260810 03:28:14.195706 13845 consensus_queue.cc:260] T 44e2930c1db3427e94f6d5b361ff8cb8 P 6420226252c2434394d828b33da15322 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30879b54184e41e184a8fa62a3bad0a4" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 41829 } } peers { permanent_uuid: "89958a9959614a45b8bcb836677733f7" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 44119 } } peers { permanent_uuid: "6420226252c2434394d828b33da15322" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 33371 } }
I20260810 03:28:14.196192 13847 raft_consensus.cc:348] T 44e2930c1db3427e94f6d5b361ff8cb8 P 30879b54184e41e184a8fa62a3bad0a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30879b54184e41e184a8fa62a3bad0a4" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 41829 } } peers { permanent_uuid: "89958a9959614a45b8bcb836677733f7" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 44119 } } peers { permanent_uuid: "6420226252c2434394d828b33da15322" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 33371 } }
I20260810 03:28:14.196769 13847 raft_consensus.cc:374] T 44e2930c1db3427e94f6d5b361ff8cb8 P 30879b54184e41e184a8fa62a3bad0a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:28:14.196983 13847 raft_consensus.cc:729] T 44e2930c1db3427e94f6d5b361ff8cb8 P 30879b54184e41e184a8fa62a3bad0a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30879b54184e41e184a8fa62a3bad0a4, State: Initialized, Role: FOLLOWER
I20260810 03:28:14.197531 13850 tablet_bootstrap.cc:492] T 44e2930c1db3427e94f6d5b361ff8cb8 P 89958a9959614a45b8bcb836677733f7: No bootstrap required, opened a new log
I20260810 03:28:14.197916 13850 ts_tablet_manager.cc:1397] T 44e2930c1db3427e94f6d5b361ff8cb8 P 89958a9959614a45b8bcb836677733f7: Time spent bootstrapping tablet: real 0.014s user 0.011s sys 0.000s
I20260810 03:28:14.198100 13594 heartbeater.cc:503] Master 127.27.161.62:32815 was elected leader, sending a full tablet report...
I20260810 03:28:14.198026 13847 consensus_queue.cc:260] T 44e2930c1db3427e94f6d5b361ff8cb8 P 30879b54184e41e184a8fa62a3bad0a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30879b54184e41e184a8fa62a3bad0a4" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 41829 } } peers { permanent_uuid: "89958a9959614a45b8bcb836677733f7" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 44119 } } peers { permanent_uuid: "6420226252c2434394d828b33da15322" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 33371 } }
I20260810 03:28:14.198814 13845 ts_tablet_manager.cc:1428] T 44e2930c1db3427e94f6d5b361ff8cb8 P 6420226252c2434394d828b33da15322: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
I20260810 03:28:14.199859 13811 heartbeater.cc:503] Master 127.27.161.62:32815 was elected leader, sending a full tablet report...
I20260810 03:28:14.200565 13847 ts_tablet_manager.cc:1428] T 44e2930c1db3427e94f6d5b361ff8cb8 P 30879b54184e41e184a8fa62a3bad0a4: Time spent starting tablet: real 0.016s user 0.013s sys 0.003s
I20260810 03:28:14.208736 13850 raft_consensus.cc:348] T 44e2930c1db3427e94f6d5b361ff8cb8 P 89958a9959614a45b8bcb836677733f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30879b54184e41e184a8fa62a3bad0a4" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 41829 } } peers { permanent_uuid: "89958a9959614a45b8bcb836677733f7" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 44119 } } peers { permanent_uuid: "6420226252c2434394d828b33da15322" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 33371 } }
I20260810 03:28:14.209183 13850 raft_consensus.cc:374] T 44e2930c1db3427e94f6d5b361ff8cb8 P 89958a9959614a45b8bcb836677733f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:28:14.209338 13850 raft_consensus.cc:729] T 44e2930c1db3427e94f6d5b361ff8cb8 P 89958a9959614a45b8bcb836677733f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89958a9959614a45b8bcb836677733f7, State: Initialized, Role: FOLLOWER
I20260810 03:28:14.209996 13850 consensus_queue.cc:260] T 44e2930c1db3427e94f6d5b361ff8cb8 P 89958a9959614a45b8bcb836677733f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30879b54184e41e184a8fa62a3bad0a4" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 41829 } } peers { permanent_uuid: "89958a9959614a45b8bcb836677733f7" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 44119 } } peers { permanent_uuid: "6420226252c2434394d828b33da15322" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 33371 } }
I20260810 03:28:14.212030 13850 ts_tablet_manager.cc:1428] T 44e2930c1db3427e94f6d5b361ff8cb8 P 89958a9959614a45b8bcb836677733f7: Time spent starting tablet: real 0.014s user 0.003s sys 0.010s
W20260810 03:28:14.275513 13364 tablet.cc:2369] T 44e2930c1db3427e94f6d5b361ff8cb8 P 89958a9959614a45b8bcb836677733f7: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 03:28:14.295573 13812 tablet.cc:2369] T 44e2930c1db3427e94f6d5b361ff8cb8 P 30879b54184e41e184a8fa62a3bad0a4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 03:28:14.363018 13595 tablet.cc:2369] T 44e2930c1db3427e94f6d5b361ff8cb8 P 6420226252c2434394d828b33da15322: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 03:28:14.382781 13855 raft_consensus.cc:482] T 44e2930c1db3427e94f6d5b361ff8cb8 P 30879b54184e41e184a8fa62a3bad0a4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 03:28:14.383208 13855 raft_consensus.cc:504] T 44e2930c1db3427e94f6d5b361ff8cb8 P 30879b54184e41e184a8fa62a3bad0a4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30879b54184e41e184a8fa62a3bad0a4" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 41829 } } peers { permanent_uuid: "89958a9959614a45b8bcb836677733f7" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 44119 } } peers { permanent_uuid: "6420226252c2434394d828b33da15322" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 33371 } }
I20260810 03:28:14.385830 13855 leader_election.cc:288] T 44e2930c1db3427e94f6d5b361ff8cb8 P 30879b54184e41e184a8fa62a3bad0a4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 89958a9959614a45b8bcb836677733f7 (127.27.161.1:44119), 6420226252c2434394d828b33da15322 (127.27.161.2:33371)
I20260810 03:28:14.393015 13839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:28:14.386250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.161.3:56879 (local address 127.27.161.1:44119)
0810 03:28:14.386523 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0810 03:28:14.386530 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:28:14.386754 (+ 224us) server_negotiation.cc:407] Connection header received
0810 03:28:14.386942 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:28:14.386947 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:28:14.387030 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:28:14.387121 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:28:14.387780 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:28:14.388931 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:28:14.391189 (+ 2258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:28:14.392077 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:28:14.392241 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:28:14.392325 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0810 03:28:14.392609 (+ 284us) server_negotiation.cc:299] Negotiation successful
0810 03:28:14.392745 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260810 03:28:14.393115 13816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:28:14.386367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.1:44119 (local address 127.27.161.3:56879)
0810 03:28:14.386557 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0810 03:28:14.386577 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 03:28:14.386737 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:28:14.387318 (+ 581us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:28:14.387323 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:28:14.387350 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:28:14.387622 (+ 272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:28:14.387637 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:28:14.389078 (+ 1441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:28:14.389082 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:28:14.391009 (+ 1927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:28:14.391024 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:28:14.392241 (+ 1217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:28:14.392247 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:28:14.392431 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:28:14.392457 (+ 26us) client_negotiation.cc:769] Sending connection context
0810 03:28:14.392553 (+ 96us) client_negotiation.cc:240] Negotiation successful
0810 03:28:14.392700 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260810 03:28:14.394492 13292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44e2930c1db3427e94f6d5b361ff8cb8" candidate_uuid: "30879b54184e41e184a8fa62a3bad0a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89958a9959614a45b8bcb836677733f7" is_pre_election: true
I20260810 03:28:14.395161 13843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:28:14.387203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.161.3:54431 (local address 127.27.161.2:33371)
0810 03:28:14.387478 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0810 03:28:14.387484 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:28:14.387949 (+ 465us) server_negotiation.cc:407] Connection header received
0810 03:28:14.388171 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:28:14.388177 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:28:14.388260 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:28:14.388355 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:28:14.389271 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:28:14.390375 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:28:14.392816 (+ 2441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:28:14.394161 (+ 1345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:28:14.394326 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:28:14.394404 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0810 03:28:14.394744 (+ 340us) server_negotiation.cc:299] Negotiation successful
0810 03:28:14.394892 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260810 03:28:14.395181 13857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:28:14.386823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.2:33371 (local address 127.27.161.3:54431)
0810 03:28:14.387812 (+ 989us) negotiation.cc:107] Waiting for socket to connect
0810 03:28:14.387837 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0810 03:28:14.387987 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:28:14.388384 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:28:14.388389 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:28:14.388414 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:28:14.389050 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:28:14.389066 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:28:14.390541 (+ 1475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:28:14.390547 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:28:14.392628 (+ 2081us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:28:14.392644 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:28:14.394370 (+ 1726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:28:14.394376 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:28:14.394562 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:28:14.394598 (+ 36us) client_negotiation.cc:769] Sending connection context
0810 03:28:14.394704 (+ 106us) client_negotiation.cc:240] Negotiation successful
0810 03:28:14.394843 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":820,"thread_start_us":661,"threads_started":1}
I20260810 03:28:14.395330 13292 raft_consensus.cc:2454] T 44e2930c1db3427e94f6d5b361ff8cb8 P 89958a9959614a45b8bcb836677733f7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30879b54184e41e184a8fa62a3bad0a4 in term 0.
I20260810 03:28:14.396286 13639 leader_election.cc:302] T 44e2930c1db3427e94f6d5b361ff8cb8 P 30879b54184e41e184a8fa62a3bad0a4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30879b54184e41e184a8fa62a3bad0a4, 89958a9959614a45b8bcb836677733f7; no voters:
I20260810 03:28:14.396381 13523 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44e2930c1db3427e94f6d5b361ff8cb8" candidate_uuid: "30879b54184e41e184a8fa62a3bad0a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6420226252c2434394d828b33da15322" is_pre_election: true
I20260810 03:28:14.396764 13855 raft_consensus.cc:2781] T 44e2930c1db3427e94f6d5b361ff8cb8 P 30879b54184e41e184a8fa62a3bad0a4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 03:28:14.396924 13855 raft_consensus.cc:482] T 44e2930c1db3427e94f6d5b361ff8cb8 P 30879b54184e41e184a8fa62a3bad0a4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 03:28:14.397012 13855 raft_consensus.cc:3037] T 44e2930c1db3427e94f6d5b361ff8cb8 P 30879b54184e41e184a8fa62a3bad0a4 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:28:14.396960 13523 raft_consensus.cc:2454] T 44e2930c1db3427e94f6d5b361ff8cb8 P 6420226252c2434394d828b33da15322 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30879b54184e41e184a8fa62a3bad0a4 in term 0.
I20260810 03:28:14.399597 13855 raft_consensus.cc:504] T 44e2930c1db3427e94f6d5b361ff8cb8 P 30879b54184e41e184a8fa62a3bad0a4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30879b54184e41e184a8fa62a3bad0a4" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 41829 } } peers { permanent_uuid: "89958a9959614a45b8bcb836677733f7" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 44119 } } peers { permanent_uuid: "6420226252c2434394d828b33da15322" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 33371 } }
I20260810 03:28:14.400207 13855 leader_election.cc:288] T 44e2930c1db3427e94f6d5b361ff8cb8 P 30879b54184e41e184a8fa62a3bad0a4 [CANDIDATE]: Term 1 election: Requested vote from peers 89958a9959614a45b8bcb836677733f7 (127.27.161.1:44119), 6420226252c2434394d828b33da15322 (127.27.161.2:33371)
I20260810 03:28:14.400744 13292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44e2930c1db3427e94f6d5b361ff8cb8" candidate_uuid: "30879b54184e41e184a8fa62a3bad0a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89958a9959614a45b8bcb836677733f7"
I20260810 03:28:14.400820 13523 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44e2930c1db3427e94f6d5b361ff8cb8" candidate_uuid: "30879b54184e41e184a8fa62a3bad0a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6420226252c2434394d828b33da15322"
I20260810 03:28:14.400954 13292 raft_consensus.cc:3037] T 44e2930c1db3427e94f6d5b361ff8cb8 P 89958a9959614a45b8bcb836677733f7 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:28:14.401029 13523 raft_consensus.cc:3037] T 44e2930c1db3427e94f6d5b361ff8cb8 P 6420226252c2434394d828b33da15322 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:28:14.403514 13523 raft_consensus.cc:2454] T 44e2930c1db3427e94f6d5b361ff8cb8 P 6420226252c2434394d828b33da15322 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30879b54184e41e184a8fa62a3bad0a4 in term 1.
I20260810 03:28:14.403710 13292 raft_consensus.cc:2454] T 44e2930c1db3427e94f6d5b361ff8cb8 P 89958a9959614a45b8bcb836677733f7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30879b54184e41e184a8fa62a3bad0a4 in term 1.
I20260810 03:28:14.403961 13639 leader_election.cc:302] T 44e2930c1db3427e94f6d5b361ff8cb8 P 30879b54184e41e184a8fa62a3bad0a4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30879b54184e41e184a8fa62a3bad0a4, 6420226252c2434394d828b33da15322; no voters:
I20260810 03:28:14.404323 13855 raft_consensus.cc:2781] T 44e2930c1db3427e94f6d5b361ff8cb8 P 30879b54184e41e184a8fa62a3bad0a4 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:28:14.404927 13855 raft_consensus.cc:686] T 44e2930c1db3427e94f6d5b361ff8cb8 P 30879b54184e41e184a8fa62a3bad0a4 [term 1 LEADER]: Becoming Leader. State: Replica: 30879b54184e41e184a8fa62a3bad0a4, State: Running, Role: LEADER
I20260810 03:28:14.405431 13855 consensus_queue.cc:237] T 44e2930c1db3427e94f6d5b361ff8cb8 P 30879b54184e41e184a8fa62a3bad0a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30879b54184e41e184a8fa62a3bad0a4" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 41829 } } peers { permanent_uuid: "89958a9959614a45b8bcb836677733f7" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 44119 } } peers { permanent_uuid: "6420226252c2434394d828b33da15322" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 33371 } }
I20260810 03:28:14.411994 13030 catalog_manager.cc:5495] T 44e2930c1db3427e94f6d5b361ff8cb8 P 30879b54184e41e184a8fa62a3bad0a4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 30879b54184e41e184a8fa62a3bad0a4 (127.27.161.3). New cstate: current_term: 1 leader_uuid: "30879b54184e41e184a8fa62a3bad0a4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30879b54184e41e184a8fa62a3bad0a4" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 41829 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "89958a9959614a45b8bcb836677733f7" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 44119 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6420226252c2434394d828b33da15322" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 33371 } health_report { overall_health: UNKNOWN } } }
I20260810 03:28:14.448532 13838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:28:14.440652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42606 (local address 127.27.161.3:41829)
0810 03:28:14.440865 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0810 03:28:14.440871 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:28:14.441839 (+ 968us) server_negotiation.cc:407] Connection header received
0810 03:28:14.442090 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:28:14.442096 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:28:14.442216 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:28:14.442381 (+ 165us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 03:28:14.442816 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:28:14.443817 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:28:14.445897 (+ 2080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:28:14.446465 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:28:14.446596 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:28:14.447244 (+ 648us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 03:28:14.447894 (+ 650us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 03:28:14.448036 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0810 03:28:14.448152 (+ 116us) server_negotiation.cc:299] Negotiation successful
0810 03:28:14.448247 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260810 03:28:14.458890 13292 raft_consensus.cc:1261] T 44e2930c1db3427e94f6d5b361ff8cb8 P 89958a9959614a45b8bcb836677733f7 [term 1 FOLLOWER]: Refusing update from remote peer 30879b54184e41e184a8fa62a3bad0a4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:28:14.458920 13523 raft_consensus.cc:1261] T 44e2930c1db3427e94f6d5b361ff8cb8 P 6420226252c2434394d828b33da15322 [term 1 FOLLOWER]: Refusing update from remote peer 30879b54184e41e184a8fa62a3bad0a4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:28:14.460216 13858 consensus_queue.cc:1035] T 44e2930c1db3427e94f6d5b361ff8cb8 P 30879b54184e41e184a8fa62a3bad0a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6420226252c2434394d828b33da15322" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 33371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:28:14.460732 13855 consensus_queue.cc:1035] T 44e2930c1db3427e94f6d5b361ff8cb8 P 30879b54184e41e184a8fa62a3bad0a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89958a9959614a45b8bcb836677733f7" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 44119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:28:14.495851 13030 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58116:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 03:28:14.496353 13030 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58116:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 03:28:14.501700 13030 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f0c134cae7974fed9683afccfc749107: Sending DeleteTablet for 3 replicas of tablet 44e2930c1db3427e94f6d5b361ff8cb8
I20260810 03:28:14.503353 13259 tablet_service.cc:1514] Processing DeleteTablet for tablet 44e2930c1db3427e94f6d5b361ff8cb8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:28:14 UTC) from {username='kudu', principal='kudu/127.27.161.62@KRBTEST.COM'} at 127.0.0.1:44374
I20260810 03:28:14.503451 13718 tablet_service.cc:1514] Processing DeleteTablet for tablet 44e2930c1db3427e94f6d5b361ff8cb8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:28:14 UTC) from {username='kudu', principal='kudu/127.27.161.62@KRBTEST.COM'} at 127.0.0.1:42604
I20260810 03:28:14.504329 13716 tablet_service.cc:1514] Processing DeleteTablet for tablet 44e2930c1db3427e94f6d5b361ff8cb8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:28:14 UTC) from {username='kudu', principal='kudu/127.27.161.62@KRBTEST.COM'} at 127.0.0.1:42604
I20260810 03:28:14.505178 13491 tablet_service.cc:1514] Processing DeleteTablet for tablet 44e2930c1db3427e94f6d5b361ff8cb8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:28:14 UTC) from {username='kudu', principal='kudu/127.27.161.62@KRBTEST.COM'} at 127.0.0.1:54526
I20260810 03:28:14.507493 28292 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iBH5/build/asan/bin/kudu pid 13155
I20260810 03:28:14.518988 13877 tablet_replica.cc:330] T 44e2930c1db3427e94f6d5b361ff8cb8 P 89958a9959614a45b8bcb836677733f7: stopping tablet replica
I20260810 03:28:14.520982 13877 raft_consensus.cc:2229] T 44e2930c1db3427e94f6d5b361ff8cb8 P 89958a9959614a45b8bcb836677733f7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 03:28:14.522307 13877 raft_consensus.cc:2258] T 44e2930c1db3427e94f6d5b361ff8cb8 P 89958a9959614a45b8bcb836677733f7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:28:14.522709 13881 tablet_replica.cc:330] T 44e2930c1db3427e94f6d5b361ff8cb8 P 6420226252c2434394d828b33da15322: stopping tablet replica
I20260810 03:28:14.523756 13878 tablet_replica.cc:330] T 44e2930c1db3427e94f6d5b361ff8cb8 P 30879b54184e41e184a8fa62a3bad0a4: stopping tablet replica
I20260810 03:28:14.524292 13878 raft_consensus.cc:2229] T 44e2930c1db3427e94f6d5b361ff8cb8 P 30879b54184e41e184a8fa62a3bad0a4 [term 1 LEADER]: Raft consensus shutting down.
I20260810 03:28:14.525198 13878 raft_consensus.cc:2258] T 44e2930c1db3427e94f6d5b361ff8cb8 P 30879b54184e41e184a8fa62a3bad0a4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:28:14.525310 13877 ts_tablet_manager.cc:1905] T 44e2930c1db3427e94f6d5b361ff8cb8 P 89958a9959614a45b8bcb836677733f7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:28:14.527263 13881 raft_consensus.cc:2229] T 44e2930c1db3427e94f6d5b361ff8cb8 P 6420226252c2434394d828b33da15322 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 03:28:14.527735 13878 ts_tablet_manager.cc:1905] T 44e2930c1db3427e94f6d5b361ff8cb8 P 30879b54184e41e184a8fa62a3bad0a4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:28:14.527976 13881 raft_consensus.cc:2258] T 44e2930c1db3427e94f6d5b361ff8cb8 P 6420226252c2434394d828b33da15322 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:28:14.533551 13877 ts_tablet_manager.cc:1918] T 44e2930c1db3427e94f6d5b361ff8cb8 P 89958a9959614a45b8bcb836677733f7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 03:28:14.538133 13878 ts_tablet_manager.cc:1918] T 44e2930c1db3427e94f6d5b361ff8cb8 P 30879b54184e41e184a8fa62a3bad0a4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 03:28:14.538563 13878 log.cc:1196] T 44e2930c1db3427e94f6d5b361ff8cb8 P 30879b54184e41e184a8fa62a3bad0a4: Deleting WAL directory at /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786332336319544-28292-0/minicluster-data/ts-2/wal/wals/44e2930c1db3427e94f6d5b361ff8cb8
I20260810 03:28:14.539247 13878 ts_tablet_manager.cc:1939] T 44e2930c1db3427e94f6d5b361ff8cb8 P 30879b54184e41e184a8fa62a3bad0a4: Deleting consensus metadata
Aug 10 03:28:14 dist-test-slave-vpvm krb5kdc[12295](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332489, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.161.1@KRBTEST.COM
I20260810 03:28:14.539626 13881 ts_tablet_manager.cc:1905] T 44e2930c1db3427e94f6d5b361ff8cb8 P 6420226252c2434394d828b33da15322: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:28:14.543017 13003 catalog_manager.cc:4841] TS 30879b54184e41e184a8fa62a3bad0a4 (127.27.161.3:41829): tablet 44e2930c1db3427e94f6d5b361ff8cb8 (table test-table [id=1bc586ac3f7c45a59f240a1d98f2db9c]) successfully deleted
W20260810 03:28:14.543545 13003 catalog_manager.cc:4808] TS 30879b54184e41e184a8fa62a3bad0a4 (127.27.161.3:41829): delete failed for tablet 44e2930c1db3427e94f6d5b361ff8cb8 because the tablet was not found. No further retry: Not found: Tablet not found: 44e2930c1db3427e94f6d5b361ff8cb8
I20260810 03:28:14.550048 13839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:28:14.509974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44382 (local address 127.27.161.1:44119)
0810 03:28:14.510213 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0810 03:28:14.510221 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 03:28:14.522074 (+ 11853us) server_negotiation.cc:407] Connection header received
0810 03:28:14.523296 (+ 1222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:28:14.523303 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:28:14.523401 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:28:14.523498 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:28:14.528478 (+ 4980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:28:14.530344 (+ 1866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:28:14.533477 (+ 3133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:28:14.534286 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:28:14.534410 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:28:14.540840 (+ 6430us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:28:14.540863 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:28:14.540881 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:28:14.540939 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:28:14.545224 (+ 4285us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:28:14.546237 (+ 1013us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:28:14.546249 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:28:14.546253 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:28:14.546439 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:28:14.546995 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:28:14.547001 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:28:14.547005 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:28:14.547585 (+ 580us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:28:14.548000 (+ 415us) server_negotiation.cc:1035] Waiting for connection context
0810 03:28:14.549210 (+ 1210us) server_negotiation.cc:299] Negotiation successful
0810 03:28:14.549499 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260810 03:28:14.553234 13877 log.cc:1196] T 44e2930c1db3427e94f6d5b361ff8cb8 P 89958a9959614a45b8bcb836677733f7: Deleting WAL directory at /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786332336319544-28292-0/minicluster-data/ts-0/wal/wals/44e2930c1db3427e94f6d5b361ff8cb8
I20260810 03:28:14.553800 13350 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:28:14.558878 13881 ts_tablet_manager.cc:1918] T 44e2930c1db3427e94f6d5b361ff8cb8 P 6420226252c2434394d828b33da15322: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 03:28:14.559118 13881 log.cc:1196] T 44e2930c1db3427e94f6d5b361ff8cb8 P 6420226252c2434394d828b33da15322: Deleting WAL directory at /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786332336319544-28292-0/minicluster-data/ts-1/wal/wals/44e2930c1db3427e94f6d5b361ff8cb8
I20260810 03:28:14.559875 13881 ts_tablet_manager.cc:1939] T 44e2930c1db3427e94f6d5b361ff8cb8 P 6420226252c2434394d828b33da15322: Deleting consensus metadata
I20260810 03:28:14.561887 13008 catalog_manager.cc:4841] TS 6420226252c2434394d828b33da15322 (127.27.161.2:33371): tablet 44e2930c1db3427e94f6d5b361ff8cb8 (table test-table [id=1bc586ac3f7c45a59f240a1d98f2db9c]) successfully deleted
I20260810 03:28:14.656064 13877 ts_tablet_manager.cc:1939] T 44e2930c1db3427e94f6d5b361ff8cb8 P 89958a9959614a45b8bcb836677733f7: Deleting consensus metadata
I20260810 03:28:14.656818 28292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iBH5/build/asan/bin/kudu with pid 13155
W20260810 03:28:14.688055 13008 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.161.1:44119 (error 108)
I20260810 03:28:14.688920 28292 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iBH5/build/asan/bin/kudu pid 13373
I20260810 03:28:14.690531 13886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:28:14.689483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.1:44119 (local address 127.0.0.1:44390)
0810 03:28:14.690208 (+ 725us) negotiation.cc:107] Waiting for socket to connect
0810 03:28:14.690335 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.1:44119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":582,"thread_start_us":607,"threads_started":1}
W20260810 03:28:14.691939 13008 catalog_manager.cc:4584] TS 89958a9959614a45b8bcb836677733f7 (127.27.161.1:44119): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 44e2930c1db3427e94f6d5b361ff8cb8: Network error: Client connection negotiation failed: client connection to 127.27.161.1:44119: connect: Connection refused (error 111)
Aug 10 03:28:14 dist-test-slave-vpvm krb5kdc[12295](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332489, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.161.2@KRBTEST.COM
I20260810 03:28:14.707301 13843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:28:14.689859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54532 (local address 127.27.161.2:33371)
0810 03:28:14.690107 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0810 03:28:14.690114 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:28:14.690137 (+ 23us) server_negotiation.cc:407] Connection header received
0810 03:28:14.690212 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:28:14.690218 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:28:14.690317 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:28:14.690445 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:28:14.691882 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:28:14.693096 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:28:14.694495 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:28:14.695235 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:28:14.695390 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:28:14.699636 (+ 4246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:28:14.699658 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:28:14.699683 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:28:14.699738 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:28:14.703724 (+ 3986us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:28:14.704574 (+ 850us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:28:14.704585 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:28:14.704589 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:28:14.704773 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:28:14.705279 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:28:14.705284 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:28:14.705287 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:28:14.705830 (+ 543us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:28:14.706182 (+ 352us) server_negotiation.cc:1035] Waiting for connection context
0810 03:28:14.706629 (+ 447us) server_negotiation.cc:299] Negotiation successful
0810 03:28:14.706877 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117}
I20260810 03:28:14.708104 13589 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:28:14.745767 13886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:28:14.745254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.1:44119 (local address 127.0.0.1:44406)
0810 03:28:14.745460 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0810 03:28:14.745582 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.1:44119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260810 03:28:14.747094 13886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:28:14.746633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.1:44119 (local address 127.0.0.1:44408)
0810 03:28:14.746821 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0810 03:28:14.746914 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.1:44119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260810 03:28:14.799697 28292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iBH5/build/asan/bin/kudu with pid 13373
I20260810 03:28:14.809762 13886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:28:14.809217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.1:44119 (local address 127.0.0.1:44422)
0810 03:28:14.809457 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0810 03:28:14.809565 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.1:44119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260810 03:28:14.814429 13886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:28:14.810463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.1:44119 (local address 127.0.0.1:44436)
0810 03:28:14.814121 (+ 3658us) negotiation.cc:107] Waiting for socket to connect
0810 03:28:14.814222 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.1:44119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":3552,"threads_started":1}
I20260810 03:28:14.829907 28292 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iBH5/build/asan/bin/kudu pid 13603
Aug 10 03:28:14 dist-test-slave-vpvm krb5kdc[12295](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332489, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.161.3@KRBTEST.COM
I20260810 03:28:14.846424 13838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:28:14.830779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42616 (local address 127.27.161.3:41829)
0810 03:28:14.830957 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0810 03:28:14.830964 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:28:14.830986 (+ 22us) server_negotiation.cc:407] Connection header received
0810 03:28:14.831207 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:28:14.831213 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:28:14.831287 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:28:14.831370 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:28:14.832762 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:28:14.833561 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:28:14.834893 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:28:14.835379 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:28:14.835503 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:28:14.840189 (+ 4686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:28:14.840205 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:28:14.840223 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:28:14.840269 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:28:14.843457 (+ 3188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:28:14.844222 (+ 765us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:28:14.844230 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:28:14.844232 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:28:14.844389 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:28:14.844755 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:28:14.844759 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:28:14.844761 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:28:14.845131 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:28:14.845443 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0810 03:28:14.845746 (+ 303us) server_negotiation.cc:299] Negotiation successful
0810 03:28:14.846104 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260810 03:28:14.847110 13803 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:28:14.896920 13888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:28:14.896338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.1:44119 (local address 127.0.0.1:44438)
0810 03:28:14.896628 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0810 03:28:14.896746 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.1:44119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260810 03:28:14.898146 13888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:28:14.897704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.1:44119 (local address 127.0.0.1:44446)
0810 03:28:14.897902 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0810 03:28:14.898028 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.1:44119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108}
I20260810 03:28:14.972770 28292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iBH5/build/asan/bin/kudu with pid 13603
I20260810 03:28:15.001937 28292 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iBH5/build/asan/bin/kudu pid 12983
I20260810 03:28:15.003237 13077 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:28:15.118358 13888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:28:15.117804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.1:44119 (local address 127.0.0.1:44448)
0810 03:28:15.118055 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0810 03:28:15.118160 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.1:44119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260810 03:28:15.118575 28292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iBH5/build/asan/bin/kudu with pid 12983
I20260810 03:28:15.153795 28292 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260810 03:28:15.499991 28292 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260810 03:28:15.500301 28292 mini_ranger.cc:66] Stopping Ranger...
I20260810 03:28:15.846884 28292 mini_ranger.cc:68] Stopped Ranger
2026-08-10 03:28:15.847 UTC [12343] LOG: received smart shutdown request
2026-08-10 03:28:15.852 UTC [12343] LOG: background worker "logical replication launcher" (PID 12359) exited with exit code 1
2026-08-10 03:28:15.853 UTC [12350] LOG: shutting down
2026-08-10 03:28:15.996 UTC [12343] LOG: database system is shut down
2026-08-10T03:28:15Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (52231 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786332336319544-28292-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 03:28:16 dist-test-slave-vpvm krb5kdc[13905](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 03:28:16 dist-test-slave-vpvm krb5kdc[13905](info): set up 2 sockets
Aug 10 03:28:16 dist-test-slave-vpvm krb5kdc[13905](info): commencing operation
krb5kdc: starting...
W20260810 03:28:17.363337 28292 mini_kdc.cc:121] Time spent starting KDC: real 1.211s user 0.000s sys 0.047s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786332336319544-28292-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786332336319544-28292-0/krb5kdc/test-user.keytab.
Aug 10 03:28:17 dist-test-slave-vpvm krb5kdc[13905](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332497, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T03:28:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T03:28:17Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.27.161.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.27.161.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.27.161.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786332336319544-28292-0/krb5kdc/rangeradmin_127.27.161.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.27.161.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786332336319544-28292-0/krb5kdc/rangeradmin_127.27.161.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.27.161.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.27.161.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.27.161.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786332336319544-28292-0/krb5kdc/rangerlookup_127.27.161.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.27.161.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786332336319544-28292-0/krb5kdc/rangerlookup_127.27.161.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.27.161.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.27.161.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.27.161.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786332336319544-28292-0/krb5kdc/HTTP_127.27.161.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.27.161.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786332336319544-28292-0/krb5kdc/HTTP_127.27.161.20@KRBTEST.COM.keytab.
I20260810 03:28:17.803114 28292 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786332336319544-28292-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task24iBH5/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786332336319544-28292-0/minicluster-data/postgres -l logfile start
2026-08-10 03:28:20.284 UTC [13957] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-10 03:28:20.284 UTC [13957] LOG: listening on IPv4 address "127.27.161.20", port 51111
2026-08-10 03:28:20.287 UTC [13957] LOG: listening on Unix socket "/tmp/.s.PGSQL.51111"
2026-08-10 03:28:20.311 UTC [13959] LOG: database system was shut down at 2026-08-10 03:28:18 UTC
2026-08-10 03:28:20.315 UTC [13957] LOG: database system is ready to accept connections
I20260810 03:28:21.423391 28292 mini_postgres.cc:96] Postgres bound to 51111
2026-08-10 03:28:21.449 UTC [13974] FATAL: database "slave" does not exist
127.27.161.20:51111 - accepting connections
I20260810 03:28:21.450399 28292 mini_ranger.cc:161] Starting Ranger...
I20260810 03:28:21.488780 28292 mini_ranger.cc:84] Created miniranger Postgres user
I20260810 03:28:21.957629 28292 mini_ranger.cc:87] Created ranger Postgres database
I20260810 03:28:21.957746 28292 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786332336319544-28292-0/minicluster-data/ranger-admin
2026-08-10 03:28:22,272 [I] DB FLAVOR :POSTGRES
2026-08-10 03:28:22,273 [I] --------- Verifying Ranger DB connection ---------
2026-08-10 03:28:22,273 [I] Checking connection..
2026-08-10 03:28:22,273 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task24iBH5/build/asan/bin/postgresql.jar:/tmp/dist-test-task24iBH5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.161.20:51111/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-10 03:28:22,564 [I] Checking connection passed.
2026-08-10 03:28:22,564 [I] --------- Verifying version history table ---------
2026-08-10 03:28:22,565 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task24iBH5/build/asan/bin/postgresql.jar:/tmp/dist-test-task24iBH5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.161.20:51111/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-10 03:28:22,856 [I] Table x_db_version_h does not exist in database ranger
2026-08-10 03:28:22,856 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task24iBH5/build/asan/bin/postgresql.jar:/tmp/dist-test-task24iBH5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.161.20:51111/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-10 03:28:23,127 [I] Table x_db_version_h does not exist in database ranger
2026-08-10 03:28:23,127 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-10 03:28:23,127 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task24iBH5/build/asan/bin/postgresql.jar:/tmp/dist-test-task24iBH5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.161.20:51111/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task24iBH5/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-10 03:28:23.397 UTC [14059] WARNING: there is no transaction in progress
2026-08-10 03:28:23,407 [I] create_dbversion_catalog.sql file imported successfully
2026-08-10 03:28:23,407 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task24iBH5/build/asan/bin/postgresql.jar:/tmp/dist-test-task24iBH5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.161.20:51111/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-10 03:28:23,687 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-10 03:28:23,687 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-10 03:28:23,687 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task24iBH5/build/asan/bin/postgresql.jar:/tmp/dist-test-task24iBH5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.161.20:51111/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-10 03:28:23,948 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task24iBH5/build/asan/bin/postgresql.jar:/tmp/dist-test-task24iBH5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.161.20:51111/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-10 03:28:24,203 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task24iBH5/build/asan/bin/postgresql.jar:/tmp/dist-test-task24iBH5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.161.20:51111/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-vpvm.c.gcp-upstream.internal','N') ;"
2026-08-10 03:28:24,451 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task24iBH5/build/asan/bin/postgresql.jar:/tmp/dist-test-task24iBH5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.161.20:51111/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-10 03:28:24,753 [I] Table x_portal_user does not exist in database ranger
2026-08-10 03:28:24,753 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task24iBH5/build/asan/bin/postgresql.jar:/tmp/dist-test-task24iBH5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.161.20:51111/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-10 03:28:25,020 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-10 03:28:25,020 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task24iBH5/build/asan/bin/postgresql.jar:/tmp/dist-test-task24iBH5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.161.20:51111/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-10 03:28:25,297 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task24iBH5/build/asan/bin/postgresql.jar:/tmp/dist-test-task24iBH5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.161.20:51111/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-10 03:28:25,571 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-10 03:28:25,571 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task24iBH5/build/asan/bin/postgresql.jar:/tmp/dist-test-task24iBH5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.161.20:51111/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task24iBH5/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-10 03:28:26.568 UTC [14233] WARNING: there is no transaction in progress
2026-08-10 03:28:26.585 UTC [14233] WARNING: there is no transaction in progress
2026-08-10 03:28:26.602 UTC [14233] WARNING: there is no transaction in progress
2026-08-10 03:28:26.619 UTC [14233] WARNING: there is no transaction in progress
2026-08-10 03:28:26.635 UTC [14233] WARNING: there is no transaction in progress
2026-08-10 03:28:26.653 UTC [14233] WARNING: there is no transaction in progress
2026-08-10 03:28:26.711 UTC [14233] WARNING: there is no transaction in progress
2026-08-10 03:28:26.718 UTC [14233] WARNING: there is no transaction in progress
2026-08-10 03:28:26.732 UTC [14233] WARNING: there is no transaction in progress
2026-08-10 03:28:26.743 UTC [14233] WARNING: there is no transaction in progress
2026-08-10 03:28:26.754 UTC [14233] WARNING: there is no transaction in progress
2026-08-10 03:28:26.770 UTC [14233] WARNING: there is no transaction in progress
2026-08-10 03:28:26.780 UTC [14233] WARNING: there is no transaction in progress
2026-08-10 03:28:26.787 UTC [14233] WARNING: there is no transaction in progress
2026-08-10 03:28:26.793 UTC [14233] WARNING: there is no transaction in progress
2026-08-10 03:28:26.799 UTC [14233] WARNING: there is no transaction in progress
2026-08-10 03:28:27,479 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-10 03:28:27,480 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task24iBH5/build/asan/bin/postgresql.jar:/tmp/dist-test-task24iBH5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.161.20:51111/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-10 03:28:27,734 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-10 03:28:27,734 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task24iBH5/build/asan/bin/postgresql.jar:/tmp/dist-test-task24iBH5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.161.20:51111/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-10 03:28:27,994 [I] Table x_portal_user already exists in database 'ranger'
2026-08-10 03:28:27,994 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task24iBH5/build/asan/bin/postgresql.jar:/tmp/dist-test-task24iBH5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.161.20:51111/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-10 03:28:28,272 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-10 03:28:28,273 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task24iBH5/build/asan/bin/postgresql.jar:/tmp/dist-test-task24iBH5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.161.20:51111/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-10 03:28:28,547 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task24iBH5/build/asan/bin/postgresql.jar:/tmp/dist-test-task24iBH5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.161.20:51111/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-10 03:28:28,822 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task24iBH5/build/asan/bin/postgresql.jar:/tmp/dist-test-task24iBH5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.161.20:51111/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-vpvm.c.gcp-upstream.internal';"
2026-08-10 03:28:29,084 [I] CORE_DB_SCHEMA import status has been updated
2026-08-10 03:28:29,084 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task24iBH5/build/asan/bin/postgresql.jar:/tmp/dist-test-task24iBH5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.161.20:51111/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-10 03:28:29,343 [I] DB_PATCHES have already been applied
I20260810 03:28:29.349023 28292 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786332336319544-28292-0/minicluster-data/ranger-admin:/tmp/dist-test-task24iBH5/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task24iBH5/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task24iBH5/build/asan/bin/postgresql.jar:/tmp/dist-test-task24iBH5/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260810 03:28:29.349124 28292 mini_ranger.cc:193] Using host: 127.27.161.20
I20260810 03:28:29.372206 28292 mini_ranger.cc:227] Ranger admin URL: http://127.27.161.20:46615
Aug 10, 2026 3:28:29 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 10, 2026 3:28:29 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 10, 2026 3:28:29 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786332336319544-28292-0/minicluster-data/ranger-admin/ews/webapp
Aug 10, 2026 3:28:29 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786332336319544-28292-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 10, 2026 3:28:29 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786332336319544-28292-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 10, 2026 3:28:30 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786332336319544-28292-0/minicluster-data/ranger-admin/ews/webapp].
Aug 10, 2026 3:28:30 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260810 03:28:30.630841 28292 mini_ranger.cc:160] Time spent starting Ranger: real 9.180s user 0.003s sys 0.074s
Aug 10, 2026 3:28:52 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 10, 2026 3:28:52 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Aug 10, 2026 3:28:52 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 10, 2026 3:28:52 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 10, 2026 3:28:52 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 10, 2026 3:28:52 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 10, 2026 3:28:52 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 10, 2026 3:28:52 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 10, 2026 3:28:52 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 10, 2026 3:28:52 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 10, 2026 3:28:52 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 10, 2026 3:28:52 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 10, 2026 3:28:52 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 10, 2026 3:28:52 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 10, 2026 3:28:52 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 10, 2026 3:28:52 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 10, 2026 3:28:52 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 10, 2026 3:28:53 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260810 03:28:54.149776 28292 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.27.161.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.27.161.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.27.161.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786332336319544-28292-0/krb5kdc/rangerkms_127.27.161.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.27.161.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786332336319544-28292-0/krb5kdc/rangerkms_127.27.161.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.27.161.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.27.161.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.27.161.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786332336319544-28292-0/krb5kdc/HTTP_127.27.161.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.27.161.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786332336319544-28292-0/krb5kdc/HTTP_127.27.161.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 10 03:28:54 dist-test-slave-vpvm krb5kdc[13905](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332534, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260810 03:28:54.393848 28292 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260810 03:28:54.422832 28292 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260810 03:28:55.191134 28292 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260810 03:28:55.191239 28292 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786332336319544-28292-0/minicluster-data/ranger-kms
I20260810 03:28:55.191272 28292 mini_ranger_kms.cc:227] Using postgres at 127.27.161.20:51111
2026-08-10 03:28:55,580 [I] DB FLAVOR :POSTGRES
2026-08-10 03:28:55,580 [I] --------- Verifying Ranger DB connection ---------
2026-08-10 03:28:55,580 [I] Checking connection
2026-08-10 03:28:55,580 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task24iBH5/build/asan/bin/postgresql.jar:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786332336319544-28292-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.161.20:51111/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-10 03:28:55,826 [I] connection success
2026-08-10 03:28:55,826 [I] --------- Verifying Ranger DB tables ---------
2026-08-10 03:28:55,826 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-10 03:28:55,827 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task24iBH5/build/asan/bin/postgresql.jar:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786332336319544-28292-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.161.20:51111/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-10 03:28:56,078 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-10 03:28:56,078 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-10 03:28:56,078 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-10 03:28:56,078 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task24iBH5/build/asan/bin/postgresql.jar:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786332336319544-28292-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.161.20:51111/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786332336319544-28292-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-10 03:28:56,380 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260810 03:28:56.836035 28292 mini_ranger_kms.cc:310] Created kms service
I20260810 03:28:57.063378 28292 mini_ranger_kms.cc:326] Created kudu user
I20260810 03:28:57.112205 28292 mini_ranger_kms.cc:343] Created rangerkms user
Aug 10 03:28:57 dist-test-slave-vpvm krb5kdc[13905](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332534, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.27.161.20@KRBTEST.COM
I20260810 03:28:57.546188 28292 mini_ranger_kms.cc:384] Added ranger policy
I20260810 03:28:57.546602 28292 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786332336319544-28292-0/minicluster-data/ranger-kms:/tmp/dist-test-task24iBH5/build/asan/bin/postgresql.jar:/tmp/dist-test-task24iBH5/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task24iBH5/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260810 03:28:57.546663 28292 mini_ranger_kms.cc:240] Using host: 127.27.161.20
I20260810 03:28:57.571895 28292 mini_ranger_kms.cc:276] Ranger KMS PID: 14512
I20260810 03:28:57.572038 28292 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.27.161.20:34575
Aug 10, 2026 3:28:58 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 10, 2026 3:28:58 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 10, 2026 3:28:58 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786332336319544-28292-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 10, 2026 3:28:58 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786332336319544-28292-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 10, 2026 3:28:58 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786332336319544-28292-0/minicluster-data/ranger-kms/ews/webapp].
Aug 10, 2026 3:28:58 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Aug 10 03:28:59 dist-test-slave-vpvm krb5kdc[13905](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786332539, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.27.161.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:29:00.236075 28292 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.842s user 0.082s sys 0.354s
I20260810 03:29:00.236366 28292 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260810 03:29:00.236620 28292 mini_ranger_kms.cc:401] 127.27.161.20:34575/kms/v1/keys
Aug 10 03:29:00 dist-test-slave-vpvm krb5kdc[13905](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332539, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.27.161.20@KRBTEST.COM for HTTP/127.27.161.20@KRBTEST.COM
Aug 10 03:29:00 dist-test-slave-vpvm krb5kdc[13905](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332539, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.27.161.20@KRBTEST.COM for HTTP/127.27.161.20@KRBTEST.COM
Aug 10 03:29:03 dist-test-slave-vpvm krb5kdc[13905](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332543, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.27.161.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.161.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.161.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.62.keytab.
Entry for principal kudu/127.27.161.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.62.keytab.
I20260810 03:29:03.329103 28292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iBH5/build/asan/bin/kudu
/tmp/dist-test-task24iBH5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786332336319544-28292-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786332336319544-28292-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786332336319544-28292-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786332336319544-28292-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.161.62:46293
--webserver_interface=127.27.161.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.62.keytab
--principal=kudu/127.27.161.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.161.20:46093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.161.62:46293
--ranger_config_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786332336319544-28292-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.161.20:34575/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786332336319544-28292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786332336319544-28292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786332336319544-28292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:29:03.692811 14593 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:29:03.693125 14593 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:29:03.693255 14593 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:29:03.700821 14593 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:29:03.700956 14593 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:29:03.701009 14593 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:29:03.701086 14593 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:29:03.701153 14593 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:29:03.711871 14593 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.161.20:46093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786332336319544-28292-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786332336319544-28292-0/minicluster-data/master-0/wal
--ranger_kms_url=127.27.161.20:34575/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.27.161.62:46293
--ranger_config_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786332336319544-28292-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.62.keytab
--principal=kudu/127.27.161.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.161.62:46293
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786332336319544-28292-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.161.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786332336319544-28292-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:29:03.713284 14593 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:29:03.714742 14593 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:29:03.726951 14607 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:29:03.727032 14610 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:29:03.726985 14603 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:29:03.728291 14593 server_base.cc:1034] running on GCE node
I20260810 03:29:03.729240 14593 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:29:03.731003 14593 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:29:03.732230 14593 hybrid_clock.cc:648] HybridClock initialized: now 1786332543732185 us; error 41 us; skew 500 ppm
I20260810 03:29:03.732627 14593 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:29:03 dist-test-slave-vpvm krb5kdc[13905](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332543, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.161.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:29:03.740032 14593 init.cc:377] Logged in from keytab as kudu/127.27.161.62@KRBTEST.COM (short username kudu)
I20260810 03:29:03.741796 14593 webserver.cc:460] Webserver started at http://127.27.161.62:43275/ using document root <none> and password file <none>
I20260810 03:29:03.742570 14593 fs_manager.cc:359] Metadata directory not provided
I20260810 03:29:03.742690 14593 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:29:03.743028 14593 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 03:29:03 dist-test-slave-vpvm krb5kdc[13905](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332543, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.161.62@KRBTEST.COM for HTTP/127.27.161.20@KRBTEST.COM
I20260810 03:29:03.804644 14593 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786332336319544-28292-0/minicluster-data/master-0/data/instance:
uuid: "2e5bc018274444ff9da6c09730cf5140"
format_stamp: "Formatted at 2026-08-10 03:29:03 on dist-test-slave-vpvm"
server_key: "d2c13e9a9fbf1da029c575d39a02f243"
server_key_iv: "51074f5207b159579244bbcd4c51c9db"
server_key_version: "kuduclusterkey@0"
I20260810 03:29:03.806047 14593 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786332336319544-28292-0/minicluster-data/master-0/wal/instance:
uuid: "2e5bc018274444ff9da6c09730cf5140"
format_stamp: "Formatted at 2026-08-10 03:29:03 on dist-test-slave-vpvm"
server_key: "d2c13e9a9fbf1da029c575d39a02f243"
server_key_iv: "51074f5207b159579244bbcd4c51c9db"
server_key_version: "kuduclusterkey@0"
I20260810 03:29:03.814584 14593 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.007s sys 0.001s
I20260810 03:29:03.838367 14622 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:29:03.840308 14593 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260810 03:29:03.840602 14593 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786332336319544-28292-0/minicluster-data/master-0/data,/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786332336319544-28292-0/minicluster-data/master-0/wal
uuid: "2e5bc018274444ff9da6c09730cf5140"
format_stamp: "Formatted at 2026-08-10 03:29:03 on dist-test-slave-vpvm"
server_key: "d2c13e9a9fbf1da029c575d39a02f243"
server_key_iv: "51074f5207b159579244bbcd4c51c9db"
server_key_version: "kuduclusterkey@0"
I20260810 03:29:03.840945 14593 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786332336319544-28292-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786332336319544-28292-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786332336319544-28292-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:29:03.875799 14593 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:29:03.881448 14593 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:29:03.881804 14593 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:29:03.901474 14593 rpc_server.cc:307] RPC server started. Bound to: 127.27.161.62:46293
I20260810 03:29:03.901489 14695 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.161.62:46293 every 8 connection(s)
I20260810 03:29:03.902988 14593 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786332336319544-28292-0/minicluster-data/master-0/data/info.pb
I20260810 03:29:03.907234 14696 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:29:03.913559 28292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iBH5/build/asan/bin/kudu as pid 14593
I20260810 03:29:03.913786 28292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786332336319544-28292-0/minicluster-data/master-0/wal/instance
I20260810 03:29:03.914718 28292 external_mini_cluster.cc:1428] Setting key f8eb14b0b595378a03ef5ff9b028d869
I20260810 03:29:03.919582 14696 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e5bc018274444ff9da6c09730cf5140: Bootstrap starting.
I20260810 03:29:03.923715 14696 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2e5bc018274444ff9da6c09730cf5140: Neither blocks nor log segments found. Creating new log.
I20260810 03:29:03.924852 14696 log.cc:824] T 00000000000000000000000000000000 P 2e5bc018274444ff9da6c09730cf5140: Log is configured to *not* fsync() on all Append() calls
I20260810 03:29:03.928109 14696 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e5bc018274444ff9da6c09730cf5140: No bootstrap required, opened a new log
Aug 10 03:29:03 dist-test-slave-vpvm krb5kdc[13905](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332543, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.161.62@KRBTEST.COM
I20260810 03:29:03.936517 14696 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2e5bc018274444ff9da6c09730cf5140 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e5bc018274444ff9da6c09730cf5140" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 46293 } }
I20260810 03:29:03.936939 14696 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2e5bc018274444ff9da6c09730cf5140 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:29:03.937083 14696 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2e5bc018274444ff9da6c09730cf5140 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e5bc018274444ff9da6c09730cf5140, State: Initialized, Role: FOLLOWER
I20260810 03:29:03.937755 14696 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2e5bc018274444ff9da6c09730cf5140 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e5bc018274444ff9da6c09730cf5140" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 46293 } }
I20260810 03:29:03.938027 14696 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2e5bc018274444ff9da6c09730cf5140 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 03:29:03.938172 14696 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2e5bc018274444ff9da6c09730cf5140 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 03:29:03.938364 14696 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2e5bc018274444ff9da6c09730cf5140 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:29:03.939059 14700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:29:03.917159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42924 (local address 127.27.161.62:46293)
0810 03:29:03.918078 (+ 919us) server_negotiation.cc:206] Beginning negotiation
0810 03:29:03.918091 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 03:29:03.918142 (+ 51us) server_negotiation.cc:407] Connection header received
0810 03:29:03.919909 (+ 1767us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:29:03.919935 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:29:03.920353 (+ 418us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:29:03.920782 (+ 429us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:29:03.922542 (+ 1760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:29:03.924013 (+ 1471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:29:03.925529 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:29:03.926341 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:29:03.926527 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:29:03.930590 (+ 4063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:29:03.930602 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:29:03.930630 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:29:03.930674 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:29:03.934230 (+ 3556us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:29:03.935086 (+ 856us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:29:03.935096 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:29:03.935103 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:29:03.935291 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:29:03.935740 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:29:03.935745 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:29:03.935747 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:29:03.936394 (+ 647us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:29:03.936726 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0810 03:29:03.937239 (+ 513us) server_negotiation.cc:299] Negotiation successful
0810 03:29:03.937746 (+ 507us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":661,"thread_start_us":416,"threads_started":1}
I20260810 03:29:03.941329 14696 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2e5bc018274444ff9da6c09730cf5140 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e5bc018274444ff9da6c09730cf5140" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 46293 } }
I20260810 03:29:03.942109 14696 leader_election.cc:302] T 00000000000000000000000000000000 P 2e5bc018274444ff9da6c09730cf5140 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2e5bc018274444ff9da6c09730cf5140; no voters:
I20260810 03:29:03.942832 14696 leader_election.cc:288] T 00000000000000000000000000000000 P 2e5bc018274444ff9da6c09730cf5140 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 03:29:03.942965 14703 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2e5bc018274444ff9da6c09730cf5140 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:29:03.943662 14703 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2e5bc018274444ff9da6c09730cf5140 [term 1 LEADER]: Becoming Leader. State: Replica: 2e5bc018274444ff9da6c09730cf5140, State: Running, Role: LEADER
I20260810 03:29:03.944205 14703 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2e5bc018274444ff9da6c09730cf5140 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e5bc018274444ff9da6c09730cf5140" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 46293 } }
I20260810 03:29:03.944942 14696 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2e5bc018274444ff9da6c09730cf5140 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:29:03.947216 14705 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e5bc018274444ff9da6c09730cf5140 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2e5bc018274444ff9da6c09730cf5140. Latest consensus state: current_term: 1 leader_uuid: "2e5bc018274444ff9da6c09730cf5140" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e5bc018274444ff9da6c09730cf5140" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 46293 } } }
I20260810 03:29:03.948248 14705 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e5bc018274444ff9da6c09730cf5140 [sys.catalog]: This master's current role is: LEADER
I20260810 03:29:03.947268 14704 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e5bc018274444ff9da6c09730cf5140 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2e5bc018274444ff9da6c09730cf5140" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e5bc018274444ff9da6c09730cf5140" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 46293 } } }
I20260810 03:29:03.948637 14704 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e5bc018274444ff9da6c09730cf5140 [sys.catalog]: This master's current role is: LEADER
I20260810 03:29:03.951999 14713 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 03:29:03.958155 14713 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 03:29:03.958899 14696 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786332336319544-28292-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260810 03:29:03.980309 14713 catalog_manager.cc:1347] Generated new cluster ID: 94a61e3e88544529b997ee24c33ed6b2
I20260810 03:29:03.980497 14713 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 03:29:04.008747 14713 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 03:29:04.010389 14713 catalog_manager.cc:1504] Loading token signing keys...
I20260810 03:29:04.019711 14713 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2e5bc018274444ff9da6c09730cf5140: Generated new TSK 0
I20260810 03:29:04.020661 14713 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.161.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.161.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.161.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.1.keytab.
Entry for principal kudu/127.27.161.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.1.keytab.
I20260810 03:29:04.889249 28292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iBH5/build/asan/bin/kudu
/tmp/dist-test-task24iBH5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786332336319544-28292-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786332336319544-28292-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786332336319544-28292-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786332336319544-28292-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.161.20:34575/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.161.1:0
--local_ip_for_outbound_sockets=127.27.161.1
--webserver_interface=127.27.161.1
--webserver_port=0
--tserver_master_addrs=127.27.161.62:46293
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.1.keytab
--principal=kudu/127.27.161.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.161.20:46093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786332336319544-28292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786332336319544-28292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786332336319544-28292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 03:29:05 dist-test-slave-vpvm krb5kdc[13905](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786332545, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.161.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260810 03:29:05.365612 14777 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:29:05.365921 14777 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:29:05.366101 14777 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:29:05.373873 14777 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:29:05.374035 14777 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:29:05.374217 14777 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.161.1
I20260810 03:29:05.385076 14777 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.161.20:46093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786332336319544-28292-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786332336319544-28292-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.27.161.20:34575/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.1.keytab
--principal=kudu/127.27.161.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.161.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786332336319544-28292-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.161.1
--webserver_port=0
--tserver_master_addrs=127.27.161.62:46293
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.161.1
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786332336319544-28292-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:29:05.386619 14777 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:29:05.388476 14777 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:29:05.413092 14790 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:29:05.413915 14791 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:29:05.415805 14795 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:29:05.417934 14777 server_base.cc:1034] running on GCE node
I20260810 03:29:05.419131 14777 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:29:05.420620 14777 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:29:05.421927 14777 hybrid_clock.cc:648] HybridClock initialized: now 1786332545421837 us; error 77 us; skew 500 ppm
I20260810 03:29:05.422519 14777 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:29:05 dist-test-slave-vpvm krb5kdc[13905](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332545, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.161.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:29:05.427521 14777 init.cc:377] Logged in from keytab as kudu/127.27.161.1@KRBTEST.COM (short username kudu)
I20260810 03:29:05.432813 14777 webserver.cc:460] Webserver started at http://127.27.161.1:35087/ using document root <none> and password file <none>
I20260810 03:29:05.433710 14777 fs_manager.cc:359] Metadata directory not provided
I20260810 03:29:05.433857 14777 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:29:05.434330 14777 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 03:29:05 dist-test-slave-vpvm krb5kdc[13905](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332545, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.161.1@KRBTEST.COM for HTTP/127.27.161.20@KRBTEST.COM
I20260810 03:29:05.455350 14777 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786332336319544-28292-0/minicluster-data/ts-0/data/instance:
uuid: "d104ff4d980646909d8d759d0d488fc8"
format_stamp: "Formatted at 2026-08-10 03:29:05 on dist-test-slave-vpvm"
server_key: "ab06e63f5ea487995feba9e86c526e68"
server_key_iv: "ea8f533243a273b792600b7902290c9e"
server_key_version: "kuduclusterkey@0"
I20260810 03:29:05.456799 14777 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786332336319544-28292-0/minicluster-data/ts-0/wal/instance:
uuid: "d104ff4d980646909d8d759d0d488fc8"
format_stamp: "Formatted at 2026-08-10 03:29:05 on dist-test-slave-vpvm"
server_key: "ab06e63f5ea487995feba9e86c526e68"
server_key_iv: "ea8f533243a273b792600b7902290c9e"
server_key_version: "kuduclusterkey@0"
I20260810 03:29:05.472132 14777 fs_manager.cc:691] Time spent creating directory manager: real 0.015s user 0.003s sys 0.004s
I20260810 03:29:05.494467 14809 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:29:05.496706 14777 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20260810 03:29:05.497033 14777 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786332336319544-28292-0/minicluster-data/ts-0/data,/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786332336319544-28292-0/minicluster-data/ts-0/wal
uuid: "d104ff4d980646909d8d759d0d488fc8"
format_stamp: "Formatted at 2026-08-10 03:29:05 on dist-test-slave-vpvm"
server_key: "ab06e63f5ea487995feba9e86c526e68"
server_key_iv: "ea8f533243a273b792600b7902290c9e"
server_key_version: "kuduclusterkey@0"
I20260810 03:29:05.497361 14777 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786332336319544-28292-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786332336319544-28292-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786332336319544-28292-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:29:05.523895 14777 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:29:05.529945 14777 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:29:05.530397 14777 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:29:05.531811 14777 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:29:05.534047 14777 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:29:05.534170 14777 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260810 03:29:05.534322 14777 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:29:05.534395 14777 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:29:05.599645 14777 rpc_server.cc:307] RPC server started. Bound to: 127.27.161.1:38941
I20260810 03:29:05.599704 14979 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.161.1:38941 every 8 connection(s)
I20260810 03:29:05.601653 14777 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786332336319544-28292-0/minicluster-data/ts-0/data/info.pb
I20260810 03:29:05.609930 28292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iBH5/build/asan/bin/kudu as pid 14777
I20260810 03:29:05.610181 28292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786332336319544-28292-0/minicluster-data/ts-0/wal/instance
I20260810 03:29:05.611021 28292 external_mini_cluster.cc:1428] Setting key 812ccc15748eadb375c183c246784442
Aug 10 03:29:05 dist-test-slave-vpvm krb5kdc[13905](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332545, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.161.1@KRBTEST.COM for kudu/127.27.161.62@KRBTEST.COM
I20260810 03:29:05.656755 14985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:29:05.607772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.161.1:40699 (local address 127.27.161.62:46293)
0810 03:29:05.630574 (+ 22802us) server_negotiation.cc:206] Beginning negotiation
0810 03:29:05.630581 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:29:05.634689 (+ 4108us) server_negotiation.cc:407] Connection header received
0810 03:29:05.634778 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:29:05.634783 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:29:05.634898 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:29:05.635007 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:29:05.637412 (+ 2405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:29:05.638629 (+ 1217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:29:05.641347 (+ 2718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:29:05.642054 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:29:05.642169 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:29:05.647552 (+ 5383us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:29:05.647562 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:29:05.647566 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:29:05.647631 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:29:05.651898 (+ 4267us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:29:05.653321 (+ 1423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:29:05.653328 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:29:05.653332 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:29:05.653512 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:29:05.654223 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:29:05.654230 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:29:05.654233 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:29:05.654672 (+ 439us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:29:05.654818 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0810 03:29:05.656042 (+ 1224us) server_negotiation.cc:299] Negotiation successful
0810 03:29:05.656317 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22603,"thread_start_us":30447,"threads_started":1}
WARNING: no policy specified for kudu/127.27.161.2@KRBTEST.COM; defaulting to no policy
I20260810 03:29:05.660081 14983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:29:05.606910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:46293 (local address 127.27.161.1:40699)
0810 03:29:05.631254 (+ 24344us) negotiation.cc:107] Waiting for socket to connect
0810 03:29:05.631342 (+ 88us) client_negotiation.cc:174] Beginning negotiation
0810 03:29:05.634070 (+ 2728us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:29:05.635515 (+ 1445us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:29:05.635531 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:29:05.636251 (+ 720us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:29:05.637161 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:29:05.637190 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:29:05.638894 (+ 1704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:29:05.638900 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:29:05.641110 (+ 2210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:29:05.641144 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:29:05.642597 (+ 1453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:29:05.642606 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:29:05.642966 (+ 360us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:29:05.643871 (+ 905us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:29:05.643905 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:29:05.647176 (+ 3271us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:29:05.652174 (+ 4998us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:29:05.652226 (+ 52us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:29:05.652251 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:29:05.653148 (+ 897us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:29:05.653695 (+ 547us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:29:05.653700 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:29:05.653704 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:29:05.654066 (+ 362us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:29:05.654967 (+ 901us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:29:05.654977 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:29:05.655506 (+ 529us) client_negotiation.cc:769] Sending connection context
0810 03:29:05.657350 (+ 1844us) client_negotiation.cc:240] Negotiation successful
0810 03:29:05.658062 (+ 712us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23968,"thread_start_us":27126,"threads_started":1}
I20260810 03:29:05.662801 14980 heartbeater.cc:347] Connected to a master server at 127.27.161.62:46293
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.161.2@KRBTEST.COM" created.
I20260810 03:29:05.663199 14980 heartbeater.cc:464] Registering TS with master...
I20260810 03:29:05.664023 14980 heartbeater.cc:511] Master 127.27.161.62:46293 requested a full tablet report, sending...
I20260810 03:29:05.666831 14653 ts_manager.cc:194] Registered new tserver with Master: d104ff4d980646909d8d759d0d488fc8 (127.27.161.1:38941)
I20260810 03:29:05.669039 14653 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.161.1@KRBTEST.COM'} at 127.27.161.1:40699
Aug 10 03:29:05 dist-test-slave-vpvm krb5kdc[13905](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332545, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.161.62@KRBTEST.COM for HTTP/127.27.161.20@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.161.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.2.keytab.
Entry for principal kudu/127.27.161.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.2.keytab.
W20260810 03:29:05.715729 28292 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.27.161.2: real 0.101s user 0.005s sys 0.024s
I20260810 03:29:05.716691 28292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iBH5/build/asan/bin/kudu
/tmp/dist-test-task24iBH5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786332336319544-28292-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786332336319544-28292-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786332336319544-28292-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786332336319544-28292-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.161.20:34575/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.161.2:0
--local_ip_for_outbound_sockets=127.27.161.2
--webserver_interface=127.27.161.2
--webserver_port=0
--tserver_master_addrs=127.27.161.62:46293
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.2.keytab
--principal=kudu/127.27.161.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.161.20:46093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786332336319544-28292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786332336319544-28292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786332336319544-28292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 03:29:05 dist-test-slave-vpvm krb5kdc[13905](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332545, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.161.62@KRBTEST.COM for HTTP/127.27.161.20@KRBTEST.COM
W20260810 03:29:06.245391 14992 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:29:06.245972 14992 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:29:06.246194 14992 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:29:06.258075 14992 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:29:06.258353 14992 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:29:06.258622 14992 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.161.2
I20260810 03:29:06.275321 14992 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.161.20:46093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786332336319544-28292-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786332336319544-28292-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.27.161.20:34575/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.2.keytab
--principal=kudu/127.27.161.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.161.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786332336319544-28292-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.161.2
--webserver_port=0
--tserver_master_addrs=127.27.161.62:46293
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.161.2
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786332336319544-28292-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:29:06.277590 14992 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:29:06.279721 14992 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:29:06.314424 15003 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:29:06.320786 15004 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:29:06.321493 14992 server_base.cc:1034] running on GCE node
W20260810 03:29:06.322638 15011 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:29:06.323729 14992 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:29:06.325302 14992 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:29:06.326632 14992 hybrid_clock.cc:648] HybridClock initialized: now 1786332546326549 us; error 69 us; skew 500 ppm
I20260810 03:29:06.327106 14992 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:29:06 dist-test-slave-vpvm krb5kdc[13905](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332546, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.161.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:29:06.332263 14992 init.cc:377] Logged in from keytab as kudu/127.27.161.2@KRBTEST.COM (short username kudu)
I20260810 03:29:06.339865 14992 webserver.cc:460] Webserver started at http://127.27.161.2:44351/ using document root <none> and password file <none>
I20260810 03:29:06.340755 14992 fs_manager.cc:359] Metadata directory not provided
I20260810 03:29:06.340917 14992 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:29:06.341295 14992 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 03:29:06 dist-test-slave-vpvm krb5kdc[13905](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332546, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.161.2@KRBTEST.COM for HTTP/127.27.161.20@KRBTEST.COM
I20260810 03:29:06.361380 14992 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786332336319544-28292-0/minicluster-data/ts-1/data/instance:
uuid: "b8be23f09895463696cbe48e639382c1"
format_stamp: "Formatted at 2026-08-10 03:29:06 on dist-test-slave-vpvm"
server_key: "9c310cb9574b667ae6438ed1eaa6c8e0"
server_key_iv: "c8ae512486f33df1299be0ccccb7907f"
server_key_version: "kuduclusterkey@0"
I20260810 03:29:06.362762 14992 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786332336319544-28292-0/minicluster-data/ts-1/wal/instance:
uuid: "b8be23f09895463696cbe48e639382c1"
format_stamp: "Formatted at 2026-08-10 03:29:06 on dist-test-slave-vpvm"
server_key: "9c310cb9574b667ae6438ed1eaa6c8e0"
server_key_iv: "c8ae512486f33df1299be0ccccb7907f"
server_key_version: "kuduclusterkey@0"
I20260810 03:29:06.371518 14992 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.002s sys 0.005s
I20260810 03:29:06.393245 15028 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:29:06.395368 14992 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260810 03:29:06.395675 14992 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786332336319544-28292-0/minicluster-data/ts-1/data,/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786332336319544-28292-0/minicluster-data/ts-1/wal
uuid: "b8be23f09895463696cbe48e639382c1"
format_stamp: "Formatted at 2026-08-10 03:29:06 on dist-test-slave-vpvm"
server_key: "9c310cb9574b667ae6438ed1eaa6c8e0"
server_key_iv: "c8ae512486f33df1299be0ccccb7907f"
server_key_version: "kuduclusterkey@0"
I20260810 03:29:06.396037 14992 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786332336319544-28292-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786332336319544-28292-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786332336319544-28292-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:29:06.425689 14992 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:29:06.431725 14992 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:29:06.432212 14992 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:29:06.433681 14992 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:29:06.435882 14992 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:29:06.436056 14992 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260810 03:29:06.436250 14992 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:29:06.436359 14992 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:29:06.485371 14992 rpc_server.cc:307] RPC server started. Bound to: 127.27.161.2:43577
I20260810 03:29:06.485395 15203 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.161.2:43577 every 8 connection(s)
I20260810 03:29:06.487434 14992 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786332336319544-28292-0/minicluster-data/ts-1/data/info.pb
I20260810 03:29:06.496183 28292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iBH5/build/asan/bin/kudu as pid 14992
I20260810 03:29:06.496384 28292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786332336319544-28292-0/minicluster-data/ts-1/wal/instance
I20260810 03:29:06.496960 28292 external_mini_cluster.cc:1428] Setting key b61b26937d614c50cc69a4fbc08ce2ca
Aug 10 03:29:06 dist-test-slave-vpvm krb5kdc[13905](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332546, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.161.2@KRBTEST.COM for kudu/127.27.161.62@KRBTEST.COM
I20260810 03:29:06.535249 15209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:29:06.491882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.161.2:37123 (local address 127.27.161.62:46293)
0810 03:29:06.509692 (+ 17810us) server_negotiation.cc:206] Beginning negotiation
0810 03:29:06.509700 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 03:29:06.509730 (+ 30us) server_negotiation.cc:407] Connection header received
0810 03:29:06.509807 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:29:06.509813 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:29:06.509922 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:29:06.510100 (+ 178us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:29:06.518173 (+ 8073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:29:06.519364 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:29:06.521370 (+ 2006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:29:06.522089 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:29:06.522255 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:29:06.526413 (+ 4158us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:29:06.526421 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:29:06.526426 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:29:06.526496 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:29:06.530375 (+ 3879us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:29:06.531573 (+ 1198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:29:06.531580 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:29:06.531583 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:29:06.531759 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:29:06.532440 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:29:06.532446 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:29:06.532449 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:29:06.532917 (+ 468us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:29:06.534366 (+ 1449us) server_negotiation.cc:1035] Waiting for connection context
0810 03:29:06.534540 (+ 174us) server_negotiation.cc:299] Negotiation successful
0810 03:29:06.534792 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17631,"thread_start_us":18328,"threads_started":1}
I20260810 03:29:06.536291 15207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:29:06.491137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:46293 (local address 127.27.161.2:37123)
0810 03:29:06.498759 (+ 7622us) negotiation.cc:107] Waiting for socket to connect
0810 03:29:06.498823 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0810 03:29:06.501570 (+ 2747us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:29:06.510558 (+ 8988us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:29:06.510574 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:29:06.511364 (+ 790us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:29:06.512293 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:29:06.512327 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:29:06.519629 (+ 7302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:29:06.519636 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:29:06.521156 (+ 1520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:29:06.521177 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:29:06.522374 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:29:06.522383 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:29:06.522679 (+ 296us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:29:06.523479 (+ 800us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:29:06.523510 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:29:06.526063 (+ 2553us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:29:06.530656 (+ 4593us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:29:06.530670 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:29:06.530692 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:29:06.531353 (+ 661us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:29:06.531923 (+ 570us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:29:06.531930 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:29:06.531933 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:29:06.532247 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:29:06.533102 (+ 855us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:29:06.533112 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:29:06.533671 (+ 559us) client_negotiation.cc:769] Sending connection context
0810 03:29:06.534070 (+ 399us) client_negotiation.cc:240] Negotiation successful
0810 03:29:06.534729 (+ 659us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7195,"thread_start_us":7479,"threads_started":1}
I20260810 03:29:06.538702 15204 heartbeater.cc:347] Connected to a master server at 127.27.161.62:46293
I20260810 03:29:06.539094 15204 heartbeater.cc:464] Registering TS with master...
I20260810 03:29:06.540136 15204 heartbeater.cc:511] Master 127.27.161.62:46293 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.27.161.3@KRBTEST.COM; defaulting to no policy
I20260810 03:29:06.542109 14653 ts_manager.cc:194] Registered new tserver with Master: b8be23f09895463696cbe48e639382c1 (127.27.161.2:43577)
I20260810 03:29:06.543025 14653 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.161.2@KRBTEST.COM'} at 127.27.161.2:37123
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.161.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.161.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.3.keytab.
Entry for principal kudu/127.27.161.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.3.keytab.
I20260810 03:29:06.579658 28292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iBH5/build/asan/bin/kudu
/tmp/dist-test-task24iBH5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786332336319544-28292-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786332336319544-28292-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786332336319544-28292-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786332336319544-28292-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.161.20:34575/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.161.3:0
--local_ip_for_outbound_sockets=127.27.161.3
--webserver_interface=127.27.161.3
--webserver_port=0
--tserver_master_addrs=127.27.161.62:46293
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.3.keytab
--principal=kudu/127.27.161.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.161.20:46093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786332336319544-28292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786332336319544-28292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786332336319544-28292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 03:29:06.672847 14980 heartbeater.cc:503] Master 127.27.161.62:46293 was elected leader, sending a full tablet report...
W20260810 03:29:06.982206 15213 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:29:06.982618 15213 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:29:06.982769 15213 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:29:06.991322 15213 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:29:06.991547 15213 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:29:06.991729 15213 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.161.3
I20260810 03:29:07.003636 15213 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.161.20:46093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786332336319544-28292-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786332336319544-28292-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.27.161.20:34575/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.3.keytab
--principal=kudu/127.27.161.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.161.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786332336319544-28292-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.161.3
--webserver_port=0
--tserver_master_addrs=127.27.161.62:46293
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.161.3
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786332336319544-28292-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:29:07.005084 15213 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:29:07.006630 15213 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:29:07.023756 15221 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:29:07.023919 15225 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:29:07.024557 15219 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:29:07.025367 15213 server_base.cc:1034] running on GCE node
I20260810 03:29:07.026121 15213 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:29:07.027329 15213 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:29:07.028612 15213 hybrid_clock.cc:648] HybridClock initialized: now 1786332547028550 us; error 47 us; skew 500 ppm
I20260810 03:29:07.028976 15213 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:29:07 dist-test-slave-vpvm krb5kdc[13905](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332547, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.161.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:29:07.033282 15213 init.cc:377] Logged in from keytab as kudu/127.27.161.3@KRBTEST.COM (short username kudu)
I20260810 03:29:07.035142 15213 webserver.cc:460] Webserver started at http://127.27.161.3:38799/ using document root <none> and password file <none>
I20260810 03:29:07.035959 15213 fs_manager.cc:359] Metadata directory not provided
I20260810 03:29:07.036198 15213 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:29:07.036664 15213 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 03:29:07 dist-test-slave-vpvm krb5kdc[13905](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332547, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.161.3@KRBTEST.COM for HTTP/127.27.161.20@KRBTEST.COM
I20260810 03:29:07.054239 15213 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786332336319544-28292-0/minicluster-data/ts-2/data/instance:
uuid: "fc38f4e60c47484abfefa958e7104dcb"
format_stamp: "Formatted at 2026-08-10 03:29:07 on dist-test-slave-vpvm"
server_key: "f4564d167ef4a53eab749c7fe9358455"
server_key_iv: "bca299b5fa4a4969a5d22c8e4528d619"
server_key_version: "kuduclusterkey@0"
I20260810 03:29:07.055313 15213 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786332336319544-28292-0/minicluster-data/ts-2/wal/instance:
uuid: "fc38f4e60c47484abfefa958e7104dcb"
format_stamp: "Formatted at 2026-08-10 03:29:07 on dist-test-slave-vpvm"
server_key: "f4564d167ef4a53eab749c7fe9358455"
server_key_iv: "bca299b5fa4a4969a5d22c8e4528d619"
server_key_version: "kuduclusterkey@0"
I20260810 03:29:07.061551 15213 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260810 03:29:07.078401 15235 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:29:07.080062 15213 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.003s
I20260810 03:29:07.080260 15213 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786332336319544-28292-0/minicluster-data/ts-2/data,/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786332336319544-28292-0/minicluster-data/ts-2/wal
uuid: "fc38f4e60c47484abfefa958e7104dcb"
format_stamp: "Formatted at 2026-08-10 03:29:07 on dist-test-slave-vpvm"
server_key: "f4564d167ef4a53eab749c7fe9358455"
server_key_iv: "bca299b5fa4a4969a5d22c8e4528d619"
server_key_version: "kuduclusterkey@0"
I20260810 03:29:07.080567 15213 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786332336319544-28292-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786332336319544-28292-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786332336319544-28292-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:29:07.095088 15213 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:29:07.100292 15213 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:29:07.100773 15213 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:29:07.102051 15213 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:29:07.104264 15213 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:29:07.104357 15213 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:29:07.104509 15213 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:29:07.104568 15213 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:29:07.153195 15213 rpc_server.cc:307] RPC server started. Bound to: 127.27.161.3:42369
I20260810 03:29:07.153259 15411 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.161.3:42369 every 8 connection(s)
I20260810 03:29:07.154738 15213 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786332336319544-28292-0/minicluster-data/ts-2/data/info.pb
I20260810 03:29:07.157210 28292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iBH5/build/asan/bin/kudu as pid 15213
I20260810 03:29:07.157387 28292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786332336319544-28292-0/minicluster-data/ts-2/wal/instance
I20260810 03:29:07.157909 28292 external_mini_cluster.cc:1428] Setting key de7c673c54de8f14815eb655c31fae7f
Aug 10 03:29:07 dist-test-slave-vpvm krb5kdc[13905](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332547, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.161.3@KRBTEST.COM for kudu/127.27.161.62@KRBTEST.COM
I20260810 03:29:07.181320 15419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:29:07.157814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.161.3:33911 (local address 127.27.161.62:46293)
0810 03:29:07.162463 (+ 4649us) server_negotiation.cc:206] Beginning negotiation
0810 03:29:07.162468 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:29:07.163206 (+ 738us) server_negotiation.cc:407] Connection header received
0810 03:29:07.165128 (+ 1922us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:29:07.165133 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:29:07.165218 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:29:07.165311 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:29:07.167143 (+ 1832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:29:07.168038 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:29:07.169609 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:29:07.170103 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:29:07.170196 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:29:07.174411 (+ 4215us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:29:07.174418 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:29:07.174421 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:29:07.174466 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:29:07.177824 (+ 3358us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:29:07.178711 (+ 887us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:29:07.178715 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:29:07.178718 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:29:07.178854 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:29:07.179328 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:29:07.179332 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:29:07.179334 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:29:07.179637 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:29:07.179783 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0810 03:29:07.180541 (+ 758us) server_negotiation.cc:299] Negotiation successful
0810 03:29:07.180807 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4531,"thread_start_us":4407,"threads_started":1}
I20260810 03:29:07.182276 15420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:29:07.158429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:46293 (local address 127.27.161.3:33911)
0810 03:29:07.163035 (+ 4606us) negotiation.cc:107] Waiting for socket to connect
0810 03:29:07.163086 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0810 03:29:07.164885 (+ 1799us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:29:07.165505 (+ 620us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:29:07.165514 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:29:07.166172 (+ 658us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:29:07.166985 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:29:07.167006 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:29:07.168156 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:29:07.168160 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:29:07.169477 (+ 1317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:29:07.169490 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:29:07.170285 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:29:07.170290 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:29:07.170496 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:29:07.171144 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:29:07.171168 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:29:07.174144 (+ 2976us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:29:07.178038 (+ 3894us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:29:07.178047 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:29:07.178063 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:29:07.178569 (+ 506us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:29:07.178971 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:29:07.178976 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:29:07.178979 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:29:07.179205 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:29:07.179773 (+ 568us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:29:07.179782 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:29:07.180157 (+ 375us) client_negotiation.cc:769] Sending connection context
0810 03:29:07.180411 (+ 254us) client_negotiation.cc:240] Negotiation successful
0810 03:29:07.180996 (+ 585us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4342,"thread_start_us":4089,"threads_started":1}
I20260810 03:29:07.183755 15415 heartbeater.cc:347] Connected to a master server at 127.27.161.62:46293
I20260810 03:29:07.184095 15415 heartbeater.cc:464] Registering TS with master...
I20260810 03:29:07.185002 15415 heartbeater.cc:511] Master 127.27.161.62:46293 requested a full tablet report, sending...
I20260810 03:29:07.186728 14653 ts_manager.cc:194] Registered new tserver with Master: fc38f4e60c47484abfefa958e7104dcb (127.27.161.3:42369)
I20260810 03:29:07.187557 14653 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.161.3@KRBTEST.COM'} at 127.27.161.3:33911
I20260810 03:29:07.199782 28292 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 03:29:07.201725 28292 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iBH5/build/asan/bin/kudu pid 14777
Aug 10 03:29:07 dist-test-slave-vpvm krb5kdc[13905](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332543, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.161.1@KRBTEST.COM
I20260810 03:29:07.221479 15423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:29:07.202810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59170 (local address 127.27.161.1:38941)
0810 03:29:07.203709 (+ 899us) server_negotiation.cc:206] Beginning negotiation
0810 03:29:07.203720 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 03:29:07.203765 (+ 45us) server_negotiation.cc:407] Connection header received
0810 03:29:07.203857 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:29:07.203882 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:29:07.204187 (+ 305us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:29:07.204343 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:29:07.206412 (+ 2069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:29:07.207618 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:29:07.209312 (+ 1694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:29:07.210091 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:29:07.210212 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:29:07.215120 (+ 4908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:29:07.215138 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:29:07.215166 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:29:07.215212 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:29:07.218162 (+ 2950us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:29:07.219048 (+ 886us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:29:07.219055 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:29:07.219058 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:29:07.219269 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:29:07.219666 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:29:07.219669 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:29:07.219671 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:29:07.220090 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:29:07.220429 (+ 339us) server_negotiation.cc:1035] Waiting for connection context
0810 03:29:07.220860 (+ 431us) server_negotiation.cc:299] Negotiation successful
0810 03:29:07.221094 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":550,"thread_start_us":366,"threads_started":1}
I20260810 03:29:07.223058 14975 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:29:07.308308 28292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iBH5/build/asan/bin/kudu with pid 14777
I20260810 03:29:07.334291 28292 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iBH5/build/asan/bin/kudu pid 14992
Aug 10 03:29:07 dist-test-slave-vpvm krb5kdc[13905](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332543, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.161.2@KRBTEST.COM
I20260810 03:29:07.358007 15426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:29:07.335303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60114 (local address 127.27.161.2:43577)
0810 03:29:07.335903 (+ 600us) server_negotiation.cc:206] Beginning negotiation
0810 03:29:07.335911 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 03:29:07.335939 (+ 28us) server_negotiation.cc:407] Connection header received
0810 03:29:07.336018 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:29:07.336025 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:29:07.336254 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:29:07.336421 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:29:07.337933 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:29:07.339090 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:29:07.340605 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:29:07.341227 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:29:07.341369 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:29:07.351025 (+ 9656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:29:07.351044 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:29:07.351063 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:29:07.351126 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:29:07.354721 (+ 3595us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:29:07.355635 (+ 914us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:29:07.355645 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:29:07.355647 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:29:07.355828 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:29:07.356283 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:29:07.356287 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:29:07.356289 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:29:07.356751 (+ 462us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:29:07.357030 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0810 03:29:07.357372 (+ 342us) server_negotiation.cc:299] Negotiation successful
0810 03:29:07.357641 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":448,"thread_start_us":284,"threads_started":1}
I20260810 03:29:07.359527 15194 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:29:07.442884 28292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iBH5/build/asan/bin/kudu with pid 14992
I20260810 03:29:07.466948 28292 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iBH5/build/asan/bin/kudu pid 15213
Aug 10 03:29:07 dist-test-slave-vpvm krb5kdc[13905](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332543, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.161.3@KRBTEST.COM
I20260810 03:29:07.487149 15430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:29:07.467993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34998 (local address 127.27.161.3:42369)
0810 03:29:07.468695 (+ 702us) server_negotiation.cc:206] Beginning negotiation
0810 03:29:07.468703 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 03:29:07.468733 (+ 30us) server_negotiation.cc:407] Connection header received
0810 03:29:07.468798 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:29:07.468805 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:29:07.469038 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:29:07.469226 (+ 188us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:29:07.470680 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:29:07.471780 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:29:07.473115 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:29:07.473711 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:29:07.473816 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:29:07.478672 (+ 4856us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:29:07.478692 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:29:07.478711 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:29:07.478763 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:29:07.482927 (+ 4164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:29:07.484121 (+ 1194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:29:07.484135 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:29:07.484137 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:29:07.484412 (+ 275us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:29:07.484983 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:29:07.484988 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:29:07.484991 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:29:07.485543 (+ 552us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:29:07.485897 (+ 354us) server_negotiation.cc:1035] Waiting for connection context
0810 03:29:07.486428 (+ 531us) server_negotiation.cc:299] Negotiation successful
0810 03:29:07.486818 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":536,"thread_start_us":356,"threads_started":1}
I20260810 03:29:07.488523 15404 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:29:07.573889 28292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iBH5/build/asan/bin/kudu with pid 15213
I20260810 03:29:07.596498 28292 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iBH5/build/asan/bin/kudu pid 14593
I20260810 03:29:07.597918 14690 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:29:07.705853 28292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iBH5/build/asan/bin/kudu with pid 14593
I20260810 03:29:07.737267 28292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iBH5/build/asan/bin/kudu
/tmp/dist-test-task24iBH5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786332336319544-28292-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786332336319544-28292-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786332336319544-28292-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786332336319544-28292-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.161.62:46293
--webserver_interface=127.27.161.62
--webserver_port=43275
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.62.keytab
--principal=kudu/127.27.161.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.161.20:46093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.161.62:46293
--ranger_config_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786332336319544-28292-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.161.20:34575/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.27.161.20:34575/kms with env {KRB5CCNAME=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786332336319544-28292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786332336319544-28292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786332336319544-28292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:29:08.106802 15442 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:29:08.107123 15442 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:29:08.107259 15442 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:29:08.114929 15442 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:29:08.115067 15442 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:29:08.115142 15442 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:29:08.115223 15442 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:29:08.115293 15442 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:29:08.126336 15442 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.161.20:46093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786332336319544-28292-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786332336319544-28292-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.27.161.20:34575/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.27.161.62:46293
--ranger_config_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786332336319544-28292-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.62.keytab
--principal=kudu/127.27.161.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.161.62:46293
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786332336319544-28292-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.161.62
--webserver_port=43275
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786332336319544-28292-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:29:08.127733 15442 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:29:08.129151 15442 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:29:08.141288 15452 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:29:08.141351 15448 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:29:08.141321 15450 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:29:08.142513 15442 server_base.cc:1034] running on GCE node
I20260810 03:29:08.143363 15442 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:29:08.144938 15442 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:29:08.146132 15442 hybrid_clock.cc:648] HybridClock initialized: now 1786332548146071 us; error 54 us; skew 500 ppm
I20260810 03:29:08.146556 15442 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:29:08 dist-test-slave-vpvm krb5kdc[13905](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332548, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.161.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:29:08.149833 15442 init.cc:377] Logged in from keytab as kudu/127.27.161.62@KRBTEST.COM (short username kudu)
I20260810 03:29:08.151733 15442 webserver.cc:460] Webserver started at http://127.27.161.62:43275/ using document root <none> and password file <none>
I20260810 03:29:08.152529 15442 fs_manager.cc:359] Metadata directory not provided
I20260810 03:29:08.152664 15442 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 10 03:29:08 dist-test-slave-vpvm krb5kdc[13905](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332548, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.161.62@KRBTEST.COM for HTTP/127.27.161.20@KRBTEST.COM
I20260810 03:29:09.016455 15442 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.004s sys 0.000s
I20260810 03:29:09.019820 15464 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:29:09.021164 15442 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260810 03:29:09.021344 15442 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786332336319544-28292-0/minicluster-data/master-0/data,/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786332336319544-28292-0/minicluster-data/master-0/wal
uuid: "2e5bc018274444ff9da6c09730cf5140"
format_stamp: "Formatted at 2026-08-10 03:29:03 on dist-test-slave-vpvm"
server_key: "d2c13e9a9fbf1da029c575d39a02f243"
server_key_iv: "51074f5207b159579244bbcd4c51c9db"
server_key_version: "kuduclusterkey@0"
I20260810 03:29:09.022379 15442 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786332336319544-28292-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786332336319544-28292-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786332336319544-28292-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:29:09.040226 15442 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:29:09.044426 15442 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:29:09.044721 15442 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:29:09.063047 15442 rpc_server.cc:307] RPC server started. Bound to: 127.27.161.62:46293
I20260810 03:29:09.063067 15554 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.161.62:46293 every 8 connection(s)
I20260810 03:29:09.064682 15442 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786332336319544-28292-0/minicluster-data/master-0/data/info.pb
I20260810 03:29:09.067202 28292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iBH5/build/asan/bin/kudu as pid 15442
I20260810 03:29:09.068071 28292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iBH5/build/asan/bin/kudu
/tmp/dist-test-task24iBH5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786332336319544-28292-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786332336319544-28292-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786332336319544-28292-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786332336319544-28292-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.161.20:34575/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.161.1:38941
--local_ip_for_outbound_sockets=127.27.161.1
--tserver_master_addrs=127.27.161.62:46293
--webserver_port=35087
--webserver_interface=127.27.161.1
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.1.keytab
--principal=kudu/127.27.161.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.161.20:46093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.27.161.20:34575/kms with env {KRB5CCNAME=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786332336319544-28292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786332336319544-28292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786332336319544-28292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 03:29:09.072407 15555 sys_catalog.cc:263] Verifying existing consensus state
I20260810 03:29:09.076784 15555 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e5bc018274444ff9da6c09730cf5140: Bootstrap starting.
I20260810 03:29:09.101994 15555 log.cc:824] T 00000000000000000000000000000000 P 2e5bc018274444ff9da6c09730cf5140: Log is configured to *not* fsync() on all Append() calls
I20260810 03:29:09.114758 15555 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e5bc018274444ff9da6c09730cf5140: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 03:29:09.115450 15555 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e5bc018274444ff9da6c09730cf5140: Bootstrap complete.
I20260810 03:29:09.128935 15555 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2e5bc018274444ff9da6c09730cf5140 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e5bc018274444ff9da6c09730cf5140" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 46293 } }
I20260810 03:29:09.130234 15555 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2e5bc018274444ff9da6c09730cf5140 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e5bc018274444ff9da6c09730cf5140, State: Initialized, Role: FOLLOWER
I20260810 03:29:09.131064 15555 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2e5bc018274444ff9da6c09730cf5140 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e5bc018274444ff9da6c09730cf5140" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 46293 } }
I20260810 03:29:09.131366 15555 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2e5bc018274444ff9da6c09730cf5140 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 03:29:09.131491 15555 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2e5bc018274444ff9da6c09730cf5140 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 03:29:09.131670 15555 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2e5bc018274444ff9da6c09730cf5140 [term 1 FOLLOWER]: Advancing to term 2
I20260810 03:29:09.136493 15555 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2e5bc018274444ff9da6c09730cf5140 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e5bc018274444ff9da6c09730cf5140" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 46293 } }
I20260810 03:29:09.137084 15555 leader_election.cc:302] T 00000000000000000000000000000000 P 2e5bc018274444ff9da6c09730cf5140 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2e5bc018274444ff9da6c09730cf5140; no voters:
I20260810 03:29:09.138078 15555 leader_election.cc:288] T 00000000000000000000000000000000 P 2e5bc018274444ff9da6c09730cf5140 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260810 03:29:09.138283 15563 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2e5bc018274444ff9da6c09730cf5140 [term 2 FOLLOWER]: Leader election won for term 2
I20260810 03:29:09.139343 15563 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2e5bc018274444ff9da6c09730cf5140 [term 2 LEADER]: Becoming Leader. State: Replica: 2e5bc018274444ff9da6c09730cf5140, State: Running, Role: LEADER
I20260810 03:29:09.140225 15555 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2e5bc018274444ff9da6c09730cf5140 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:29:09.140051 15563 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2e5bc018274444ff9da6c09730cf5140 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e5bc018274444ff9da6c09730cf5140" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 46293 } }
I20260810 03:29:09.143150 15566 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e5bc018274444ff9da6c09730cf5140 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2e5bc018274444ff9da6c09730cf5140. Latest consensus state: current_term: 2 leader_uuid: "2e5bc018274444ff9da6c09730cf5140" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e5bc018274444ff9da6c09730cf5140" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 46293 } } }
I20260810 03:29:09.144111 15566 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e5bc018274444ff9da6c09730cf5140 [sys.catalog]: This master's current role is: LEADER
I20260810 03:29:09.143745 15565 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e5bc018274444ff9da6c09730cf5140 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "2e5bc018274444ff9da6c09730cf5140" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e5bc018274444ff9da6c09730cf5140" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 46293 } } }
I20260810 03:29:09.144646 15565 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e5bc018274444ff9da6c09730cf5140 [sys.catalog]: This master's current role is: LEADER
I20260810 03:29:09.151090 15575 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 03:29:09.158303 15575 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 03:29:09.160023 15575 catalog_manager.cc:1259] Loaded cluster ID: 94a61e3e88544529b997ee24c33ed6b2
I20260810 03:29:09.160209 15575 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 03:29:09.164350 15575 catalog_manager.cc:1504] Loading token signing keys...
I20260810 03:29:09.166092 15575 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 2e5bc018274444ff9da6c09730cf5140: Loaded TSK: 0
I20260810 03:29:09.167861 15575 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 03:29:09.175434 15555 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786332336319544-28292-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
W20260810 03:29:09.581239 15558 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:29:09.581749 15558 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:29:09.581974 15558 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:29:09.594010 15558 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:29:09.594246 15558 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:29:09.594483 15558 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.161.1
I20260810 03:29:09.611001 15558 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.161.20:46093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786332336319544-28292-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786332336319544-28292-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.27.161.20:34575/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.1.keytab
--principal=kudu/127.27.161.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.161.1:38941
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786332336319544-28292-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.161.1
--webserver_port=35087
--tserver_master_addrs=127.27.161.62:46293
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.161.1
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786332336319544-28292-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:29:09.613090 15558 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:29:09.615048 15558 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:29:09.639452 15612 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:29:09.640748 15613 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:29:09.648782 15558 server_base.cc:1034] running on GCE node
W20260810 03:29:09.650565 15617 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:29:09.652519 15558 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:29:09.653913 15558 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:29:09.655201 15558 hybrid_clock.cc:648] HybridClock initialized: now 1786332549655138 us; error 57 us; skew 500 ppm
I20260810 03:29:09.655571 15558 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:29:09 dist-test-slave-vpvm krb5kdc[13905](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332549, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.161.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:29:09.659137 15558 init.cc:377] Logged in from keytab as kudu/127.27.161.1@KRBTEST.COM (short username kudu)
I20260810 03:29:09.663404 15558 webserver.cc:460] Webserver started at http://127.27.161.1:35087/ using document root <none> and password file <none>
I20260810 03:29:09.664446 15558 fs_manager.cc:359] Metadata directory not provided
I20260810 03:29:09.664615 15558 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 10 03:29:09 dist-test-slave-vpvm krb5kdc[13905](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332549, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.161.1@KRBTEST.COM for HTTP/127.27.161.20@KRBTEST.COM
I20260810 03:29:09.726377 15558 fs_manager.cc:709] Time spent opening directory manager: real 0.023s user 0.006s sys 0.004s
I20260810 03:29:09.740967 15632 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:29:09.743325 15558 fs_manager.cc:725] Time spent opening block manager: real 0.015s user 0.006s sys 0.008s
I20260810 03:29:09.743577 15558 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786332336319544-28292-0/minicluster-data/ts-0/data,/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786332336319544-28292-0/minicluster-data/ts-0/wal
uuid: "d104ff4d980646909d8d759d0d488fc8"
format_stamp: "Formatted at 2026-08-10 03:29:05 on dist-test-slave-vpvm"
server_key: "ab06e63f5ea487995feba9e86c526e68"
server_key_iv: "ea8f533243a273b792600b7902290c9e"
server_key_version: "kuduclusterkey@0"
I20260810 03:29:09.745358 15558 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786332336319544-28292-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786332336319544-28292-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786332336319544-28292-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:29:09.801826 15558 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:29:09.807377 15558 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:29:09.807709 15558 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:29:09.809094 15558 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:29:09.811105 15558 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:29:09.811228 15558 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:29:09.811343 15558 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:29:09.811426 15558 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:29:09.892431 15558 rpc_server.cc:307] RPC server started. Bound to: 127.27.161.1:38941
I20260810 03:29:09.892495 15817 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.161.1:38941 every 8 connection(s)
I20260810 03:29:09.894243 15558 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786332336319544-28292-0/minicluster-data/ts-0/data/info.pb
I20260810 03:29:09.899062 28292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iBH5/build/asan/bin/kudu as pid 15558
I20260810 03:29:09.899920 28292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iBH5/build/asan/bin/kudu
/tmp/dist-test-task24iBH5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786332336319544-28292-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786332336319544-28292-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786332336319544-28292-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786332336319544-28292-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.161.20:34575/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.161.2:43577
--local_ip_for_outbound_sockets=127.27.161.2
--tserver_master_addrs=127.27.161.62:46293
--webserver_port=44351
--webserver_interface=127.27.161.2
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.2.keytab
--principal=kudu/127.27.161.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.161.20:46093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.27.161.20:34575/kms with env {KRB5CCNAME=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786332336319544-28292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786332336319544-28292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786332336319544-28292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 03:29:09 dist-test-slave-vpvm krb5kdc[13905](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332549, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.161.1@KRBTEST.COM for kudu/127.27.161.62@KRBTEST.COM
I20260810 03:29:09.955068 15827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:29:09.898062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.161.1:56913 (local address 127.27.161.62:46293)
0810 03:29:09.926109 (+ 28047us) server_negotiation.cc:206] Beginning negotiation
0810 03:29:09.926147 (+ 38us) server_negotiation.cc:399] Waiting for connection header
0810 03:29:09.927635 (+ 1488us) server_negotiation.cc:407] Connection header received
0810 03:29:09.930370 (+ 2735us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:29:09.930390 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:29:09.930955 (+ 565us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:29:09.931400 (+ 445us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:29:09.934084 (+ 2684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:29:09.936211 (+ 2127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:29:09.938842 (+ 2631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:29:09.939942 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:29:09.940116 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:29:09.945312 (+ 5196us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:29:09.945325 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:29:09.945348 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:29:09.945409 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:29:09.949609 (+ 4200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:29:09.950628 (+ 1019us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:29:09.950637 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:29:09.950643 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:29:09.950832 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:29:09.951348 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:29:09.951354 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:29:09.951357 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:29:09.951992 (+ 635us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:29:09.952205 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0810 03:29:09.953057 (+ 852us) server_negotiation.cc:299] Negotiation successful
0810 03:29:09.953445 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27600,"thread_start_us":27975,"threads_started":1}
I20260810 03:29:09.962751 15829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:29:09.898086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:46293 (local address 127.27.161.1:56913)
0810 03:29:09.927077 (+ 28991us) negotiation.cc:107] Waiting for socket to connect
0810 03:29:09.927151 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0810 03:29:09.929597 (+ 2446us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:29:09.932209 (+ 2612us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:29:09.932222 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:29:09.932945 (+ 723us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:29:09.933766 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:29:09.933795 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:29:09.936490 (+ 2695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:29:09.936496 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:29:09.938643 (+ 2147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:29:09.938663 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:29:09.940699 (+ 2036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:29:09.940708 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:29:09.941110 (+ 402us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:29:09.942001 (+ 891us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:29:09.942042 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:29:09.944934 (+ 2892us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:29:09.949828 (+ 4894us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:29:09.949837 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:29:09.949851 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:29:09.950449 (+ 598us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:29:09.950966 (+ 517us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:29:09.950971 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:29:09.950975 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:29:09.951215 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:29:09.952154 (+ 939us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:29:09.952162 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:29:09.952522 (+ 360us) client_negotiation.cc:769] Sending connection context
0810 03:29:09.952799 (+ 277us) client_negotiation.cc:240] Negotiation successful
0810 03:29:09.953445 (+ 646us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28683,"thread_start_us":28441,"threads_started":1}
I20260810 03:29:09.966308 15818 heartbeater.cc:347] Connected to a master server at 127.27.161.62:46293
I20260810 03:29:09.966663 15818 heartbeater.cc:464] Registering TS with master...
I20260810 03:29:09.967490 15818 heartbeater.cc:511] Master 127.27.161.62:46293 requested a full tablet report, sending...
W20260810 03:29:09.971184 15818 heartbeater.cc:644] Failed to heartbeat to 127.27.161.62:46293 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260810 03:29:09.971454 15818 heartbeater.cc:464] Registering TS with master...
I20260810 03:29:09.971772 15818 heartbeater.cc:511] Master 127.27.161.62:46293 requested a full tablet report, sending...
I20260810 03:29:09.973095 15818 heartbeater.cc:464] Registering TS with master...
I20260810 03:29:09.973402 15818 heartbeater.cc:511] Master 127.27.161.62:46293 requested a full tablet report, sending...
W20260810 03:29:09.974650 15818 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260810 03:29:10.411742 15831 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:29:10.412211 15831 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:29:10.412443 15831 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:29:10.424710 15831 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:29:10.424887 15831 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:29:10.425146 15831 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.161.2
I20260810 03:29:10.441409 15831 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.161.20:46093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786332336319544-28292-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786332336319544-28292-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.27.161.20:34575/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.2.keytab
--principal=kudu/127.27.161.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.161.2:43577
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786332336319544-28292-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.161.2
--webserver_port=44351
--tserver_master_addrs=127.27.161.62:46293
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.161.2
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786332336319544-28292-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:29:10.443571 15831 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:29:10.445484 15831 file_cache.cc:493] Constructed file cache file cache with capacity 419430
Aug 10 03:29:10 dist-test-slave-vpvm krb5kdc[13905](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786332550, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.161.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260810 03:29:10.465340 15857 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:29:10.466337 15860 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:29:10.486459 15862 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:29:10.490545 15831 server_base.cc:1034] running on GCE node
I20260810 03:29:10.491606 15831 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:29:10.493026 15831 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:29:10.494318 15831 hybrid_clock.cc:648] HybridClock initialized: now 1786332550494264 us; error 52 us; skew 500 ppm
I20260810 03:29:10.494684 15831 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:29:10 dist-test-slave-vpvm krb5kdc[13905](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332550, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.161.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:29:10.499060 15831 init.cc:377] Logged in from keytab as kudu/127.27.161.2@KRBTEST.COM (short username kudu)
I20260810 03:29:10.510478 15831 webserver.cc:460] Webserver started at http://127.27.161.2:44351/ using document root <none> and password file <none>
I20260810 03:29:10.511375 15831 fs_manager.cc:359] Metadata directory not provided
I20260810 03:29:10.511543 15831 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 10 03:29:10 dist-test-slave-vpvm krb5kdc[13905](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332550, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.161.2@KRBTEST.COM for HTTP/127.27.161.20@KRBTEST.COM
I20260810 03:29:10.535286 15831 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260810 03:29:10.538524 15876 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:29:10.540546 15831 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20260810 03:29:10.540768 15831 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786332336319544-28292-0/minicluster-data/ts-1/data,/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786332336319544-28292-0/minicluster-data/ts-1/wal
uuid: "b8be23f09895463696cbe48e639382c1"
format_stamp: "Formatted at 2026-08-10 03:29:06 on dist-test-slave-vpvm"
server_key: "9c310cb9574b667ae6438ed1eaa6c8e0"
server_key_iv: "c8ae512486f33df1299be0ccccb7907f"
server_key_version: "kuduclusterkey@0"
I20260810 03:29:10.542439 15831 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786332336319544-28292-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786332336319544-28292-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786332336319544-28292-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:29:10.567893 15831 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:29:10.573689 15831 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:29:10.574079 15831 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:29:10.575507 15831 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:29:10.577415 15831 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:29:10.577553 15831 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:29:10.577687 15831 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:29:10.577780 15831 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:29:10.617066 15831 rpc_server.cc:307] RPC server started. Bound to: 127.27.161.2:43577
I20260810 03:29:10.617116 16060 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.161.2:43577 every 8 connection(s)
I20260810 03:29:10.619167 15831 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786332336319544-28292-0/minicluster-data/ts-1/data/info.pb
I20260810 03:29:10.625605 28292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iBH5/build/asan/bin/kudu as pid 15831
I20260810 03:29:10.626791 28292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iBH5/build/asan/bin/kudu
/tmp/dist-test-task24iBH5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786332336319544-28292-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786332336319544-28292-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786332336319544-28292-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786332336319544-28292-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.161.20:34575/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.161.3:42369
--local_ip_for_outbound_sockets=127.27.161.3
--tserver_master_addrs=127.27.161.62:46293
--webserver_port=38799
--webserver_interface=127.27.161.3
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.3.keytab
--principal=kudu/127.27.161.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.161.20:46093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.27.161.20:34575/kms with env {KRB5CCNAME=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786332336319544-28292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786332336319544-28292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786332336319544-28292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 03:29:10 dist-test-slave-vpvm krb5kdc[13905](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332550, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.161.2@KRBTEST.COM for kudu/127.27.161.62@KRBTEST.COM
I20260810 03:29:10.652395 16063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:29:10.623277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.161.2:47537 (local address 127.27.161.62:46293)
0810 03:29:10.627366 (+ 4089us) server_negotiation.cc:206] Beginning negotiation
0810 03:29:10.627373 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:29:10.628705 (+ 1332us) server_negotiation.cc:407] Connection header received
0810 03:29:10.631400 (+ 2695us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:29:10.631407 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:29:10.631512 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:29:10.631656 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:29:10.633815 (+ 2159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:29:10.634879 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:29:10.637261 (+ 2382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:29:10.637810 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:29:10.637912 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:29:10.643294 (+ 5382us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:29:10.643303 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:29:10.643309 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:29:10.643361 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:29:10.647309 (+ 3948us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:29:10.648681 (+ 1372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:29:10.648688 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:29:10.648691 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:29:10.648910 (+ 219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:29:10.649683 (+ 773us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:29:10.649688 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:29:10.649692 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:29:10.650236 (+ 544us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:29:10.650384 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0810 03:29:10.651675 (+ 1291us) server_negotiation.cc:299] Negotiation successful
0810 03:29:10.651970 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3656,"thread_start_us":2990,"threads_started":1}
I20260810 03:29:10.655833 16064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:29:10.623898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:46293 (local address 127.27.161.2:47537)
0810 03:29:10.628476 (+ 4578us) negotiation.cc:107] Waiting for socket to connect
0810 03:29:10.628543 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0810 03:29:10.630979 (+ 2436us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:29:10.632007 (+ 1028us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:29:10.632022 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:29:10.632652 (+ 630us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:29:10.633572 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:29:10.633601 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:29:10.635130 (+ 1529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:29:10.635135 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:29:10.637059 (+ 1924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:29:10.637078 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:29:10.638358 (+ 1280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:29:10.638365 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:29:10.638678 (+ 313us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:29:10.639504 (+ 826us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:29:10.639538 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:29:10.642900 (+ 3362us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:29:10.647568 (+ 4668us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:29:10.647581 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:29:10.647606 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:29:10.648500 (+ 894us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:29:10.649162 (+ 662us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:29:10.649167 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:29:10.649171 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:29:10.649524 (+ 353us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:29:10.650512 (+ 988us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:29:10.650521 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:29:10.651040 (+ 519us) client_negotiation.cc:769] Sending connection context
0810 03:29:10.653078 (+ 2038us) client_negotiation.cc:240] Negotiation successful
0810 03:29:10.654036 (+ 958us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3804,"thread_start_us":3555,"threads_started":1}
I20260810 03:29:10.658461 16061 heartbeater.cc:347] Connected to a master server at 127.27.161.62:46293
I20260810 03:29:10.658867 16061 heartbeater.cc:464] Registering TS with master...
I20260810 03:29:10.659871 16061 heartbeater.cc:511] Master 127.27.161.62:46293 requested a full tablet report, sending...
I20260810 03:29:10.662415 15487 ts_manager.cc:194] Registered new tserver with Master: b8be23f09895463696cbe48e639382c1 (127.27.161.2:43577)
I20260810 03:29:10.664935 15487 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.161.2@KRBTEST.COM'} at 127.27.161.2:47537
Aug 10 03:29:10 dist-test-slave-vpvm krb5kdc[13905](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332550, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.161.62@KRBTEST.COM for HTTP/127.27.161.20@KRBTEST.COM
I20260810 03:29:10.974802 15818 heartbeater.cc:464] Registering TS with master...
I20260810 03:29:10.975284 15818 heartbeater.cc:511] Master 127.27.161.62:46293 requested a full tablet report, sending...
I20260810 03:29:10.982546 15487 ts_manager.cc:194] Registered new tserver with Master: d104ff4d980646909d8d759d0d488fc8 (127.27.161.1:38941)
I20260810 03:29:10.983724 15487 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.161.1@KRBTEST.COM'} at 127.27.161.1:56913
W20260810 03:29:11.110335 16065 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:29:11.110812 16065 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:29:11.110966 16065 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:29:11.119056 16065 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:29:11.119192 16065 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:29:11.119349 16065 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.161.3
I20260810 03:29:11.129814 16065 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.161.20:46093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786332336319544-28292-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786332336319544-28292-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.27.161.20:34575/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.3.keytab
--principal=kudu/127.27.161.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.161.3:42369
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786332336319544-28292-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.161.3
--webserver_port=38799
--tserver_master_addrs=127.27.161.62:46293
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.161.3
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786332336319544-28292-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:29:11.131302 16065 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:29:11.132695 16065 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:29:11.149693 16071 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:29:11.151707 16076 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:29:11.158849 16078 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:29:11.160167 16065 server_base.cc:1034] running on GCE node
I20260810 03:29:11.160861 16065 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:29:11.162228 16065 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:29:11.163570 16065 hybrid_clock.cc:648] HybridClock initialized: now 1786332551163493 us; error 62 us; skew 500 ppm
I20260810 03:29:11.164052 16065 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:29:11 dist-test-slave-vpvm krb5kdc[13905](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332551, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.161.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:29:11.167459 16065 init.cc:377] Logged in from keytab as kudu/127.27.161.3@KRBTEST.COM (short username kudu)
I20260810 03:29:11.171446 16065 webserver.cc:460] Webserver started at http://127.27.161.3:38799/ using document root <none> and password file <none>
I20260810 03:29:11.172302 16065 fs_manager.cc:359] Metadata directory not provided
I20260810 03:29:11.172453 16065 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 10 03:29:11 dist-test-slave-vpvm krb5kdc[13905](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332551, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.161.3@KRBTEST.COM for HTTP/127.27.161.20@KRBTEST.COM
I20260810 03:29:11.202353 16065 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.005s sys 0.000s
I20260810 03:29:11.213217 16090 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:29:11.215106 16065 fs_manager.cc:725] Time spent opening block manager: real 0.011s user 0.005s sys 0.004s
I20260810 03:29:11.215319 16065 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786332336319544-28292-0/minicluster-data/ts-2/data,/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786332336319544-28292-0/minicluster-data/ts-2/wal
uuid: "fc38f4e60c47484abfefa958e7104dcb"
format_stamp: "Formatted at 2026-08-10 03:29:07 on dist-test-slave-vpvm"
server_key: "f4564d167ef4a53eab749c7fe9358455"
server_key_iv: "bca299b5fa4a4969a5d22c8e4528d619"
server_key_version: "kuduclusterkey@0"
I20260810 03:29:11.216415 16065 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786332336319544-28292-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786332336319544-28292-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786332336319544-28292-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:29:11.232455 16065 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:29:11.238189 16065 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:29:11.238624 16065 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:29:11.249194 16065 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:29:11.258157 16065 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:29:11.258347 16065 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:29:11.258481 16065 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:29:11.258595 16065 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 03:29:11 dist-test-slave-vpvm krb5kdc[13905](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332550, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.161.62@KRBTEST.COM for HTTP/127.27.161.20@KRBTEST.COM
I20260810 03:29:11.499979 16065 rpc_server.cc:307] RPC server started. Bound to: 127.27.161.3:42369
I20260810 03:29:11.500030 16287 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.161.3:42369 every 8 connection(s)
I20260810 03:29:11.502512 16065 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786332336319544-28292-0/minicluster-data/ts-2/data/info.pb
I20260810 03:29:11.510941 28292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iBH5/build/asan/bin/kudu as pid 16065
Aug 10 03:29:11 dist-test-slave-vpvm krb5kdc[13905](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332551, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.161.3@KRBTEST.COM for kudu/127.27.161.62@KRBTEST.COM
I20260810 03:29:11.555080 16292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:29:11.507478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.161.3:40007 (local address 127.27.161.62:46293)
0810 03:29:11.522934 (+ 15456us) server_negotiation.cc:206] Beginning negotiation
0810 03:29:11.522940 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:29:11.522962 (+ 22us) server_negotiation.cc:407] Connection header received
0810 03:29:11.525904 (+ 2942us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:29:11.525912 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:29:11.526089 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:29:11.526204 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:29:11.529177 (+ 2973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:29:11.530516 (+ 1339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:29:11.533423 (+ 2907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:29:11.534261 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:29:11.534378 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:29:11.546450 (+ 12072us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:29:11.546457 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:29:11.546462 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:29:11.546535 (+ 73us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:29:11.550708 (+ 4173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:29:11.551850 (+ 1142us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:29:11.551857 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:29:11.551861 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:29:11.551997 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:29:11.552633 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:29:11.552643 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:29:11.552646 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:29:11.553044 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:29:11.553176 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0810 03:29:11.554381 (+ 1205us) server_negotiation.cc:299] Negotiation successful
0810 03:29:11.554672 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15342,"thread_start_us":15217,"threads_started":1}
I20260810 03:29:11.558017 16291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:29:11.506727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:46293 (local address 127.27.161.3:40007)
0810 03:29:11.509257 (+ 2530us) negotiation.cc:107] Waiting for socket to connect
0810 03:29:11.509316 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0810 03:29:11.525218 (+ 15902us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:29:11.526832 (+ 1614us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:29:11.526849 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:29:11.527617 (+ 768us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:29:11.528842 (+ 1225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:29:11.528884 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:29:11.530817 (+ 1933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:29:11.530825 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:29:11.533168 (+ 2343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:29:11.533191 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:29:11.534842 (+ 1651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:29:11.534851 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:29:11.535249 (+ 398us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:29:11.536717 (+ 1468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:29:11.536765 (+ 48us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:29:11.543287 (+ 6522us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:29:11.550921 (+ 7634us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:29:11.550934 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:29:11.550957 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:29:11.551723 (+ 766us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:29:11.552137 (+ 414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:29:11.552142 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:29:11.552146 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:29:11.552471 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:29:11.553311 (+ 840us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:29:11.553321 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:29:11.553864 (+ 543us) client_negotiation.cc:769] Sending connection context
0810 03:29:11.555691 (+ 1827us) client_negotiation.cc:240] Negotiation successful
0810 03:29:11.556374 (+ 683us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2176,"thread_start_us":1886,"threads_started":1}
I20260810 03:29:11.559690 16297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:29:11.522912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57106 (local address 127.27.161.62:46293)
0810 03:29:11.536180 (+ 13268us) server_negotiation.cc:206] Beginning negotiation
0810 03:29:11.536187 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:29:11.536208 (+ 21us) server_negotiation.cc:407] Connection header received
0810 03:29:11.536265 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:29:11.536270 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:29:11.536365 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:29:11.540333 (+ 3968us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:29:11.540654 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:29:11.542283 (+ 1629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:29:11.545409 (+ 3126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:29:11.546090 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:29:11.547571 (+ 1481us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:29:11.550912 (+ 3341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:29:11.550920 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:29:11.550924 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:29:11.550970 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:29:11.554759 (+ 3789us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:29:11.555820 (+ 1061us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:29:11.555825 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:29:11.555828 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:29:11.555980 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:29:11.556482 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:29:11.556486 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:29:11.556489 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:29:11.556873 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:29:11.558463 (+ 1590us) server_negotiation.cc:1035] Waiting for connection context
0810 03:29:11.559069 (+ 606us) server_negotiation.cc:299] Negotiation successful
0810 03:29:11.559331 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13121,"thread_start_us":12986,"threads_started":1}
I20260810 03:29:11.560050 16288 heartbeater.cc:347] Connected to a master server at 127.27.161.62:46293
I20260810 03:29:11.560380 16288 heartbeater.cc:464] Registering TS with master...
I20260810 03:29:11.561203 16288 heartbeater.cc:511] Master 127.27.161.62:46293 requested a full tablet report, sending...
I20260810 03:29:11.563391 15486 ts_manager.cc:194] Registered new tserver with Master: fc38f4e60c47484abfefa958e7104dcb (127.27.161.3:42369)
I20260810 03:29:11.564812 15486 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.161.3@KRBTEST.COM'} at 127.27.161.3:40007
I20260810 03:29:11.565279 28292 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 03:29:11.585711 16297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:29:11.571494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57108 (local address 127.27.161.62:46293)
0810 03:29:11.571715 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0810 03:29:11.571722 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:29:11.571846 (+ 124us) server_negotiation.cc:407] Connection header received
0810 03:29:11.572129 (+ 283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:29:11.572136 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:29:11.572212 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:29:11.572301 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:29:11.573880 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:29:11.574778 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:29:11.576209 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:29:11.576695 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:29:11.576818 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:29:11.579506 (+ 2688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:29:11.579511 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:29:11.579515 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:29:11.579555 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:29:11.582417 (+ 2862us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:29:11.583165 (+ 748us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:29:11.583171 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:29:11.583175 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:29:11.583352 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:29:11.583831 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:29:11.583836 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:29:11.583840 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:29:11.584205 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:29:11.584460 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0810 03:29:11.585088 (+ 628us) server_negotiation.cc:299] Negotiation successful
0810 03:29:11.585344 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260810 03:29:11.592533 15486 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57108:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 03:29:11.604307 15486 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 03:29:11.648761 16332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:29:11.622528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.3:42369 (local address 127.0.0.1:43492)
0810 03:29:11.630586 (+ 8058us) negotiation.cc:107] Waiting for socket to connect
0810 03:29:11.630645 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0810 03:29:11.630841 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:29:11.638580 (+ 7739us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:29:11.638589 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:29:11.638635 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:29:11.639166 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:29:11.639183 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:29:11.641702 (+ 2519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:29:11.641708 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:29:11.644375 (+ 2667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:29:11.644393 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:29:11.647528 (+ 3135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:29:11.647539 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:29:11.647770 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:29:11.647846 (+ 76us) client_negotiation.cc:769] Sending connection context
0810 03:29:11.647967 (+ 121us) client_negotiation.cc:240] Negotiation successful
0810 03:29:11.648187 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7743,"thread_start_us":15422,"threads_started":1}
I20260810 03:29:11.650525 16338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:29:11.623942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43492 (local address 127.27.161.3:42369)
0810 03:29:11.636349 (+ 12407us) server_negotiation.cc:206] Beginning negotiation
0810 03:29:11.636357 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 03:29:11.636395 (+ 38us) server_negotiation.cc:407] Connection header received
0810 03:29:11.636510 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:29:11.636519 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:29:11.636860 (+ 341us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:29:11.639393 (+ 2533us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:29:11.639748 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:29:11.641484 (+ 1736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:29:11.644617 (+ 3133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:29:11.647124 (+ 2507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:29:11.649357 (+ 2233us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:29:11.649788 (+ 431us) server_negotiation.cc:1035] Waiting for connection context
0810 03:29:11.649881 (+ 93us) server_negotiation.cc:299] Negotiation successful
0810 03:29:11.650153 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12229,"thread_start_us":12076,"threads_started":1}
I20260810 03:29:11.651552 16335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:29:11.624005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52122 (local address 127.27.161.2:43577)
0810 03:29:11.635844 (+ 11839us) server_negotiation.cc:206] Beginning negotiation
0810 03:29:11.635853 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 03:29:11.638370 (+ 2517us) server_negotiation.cc:407] Connection header received
0810 03:29:11.638532 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:29:11.638542 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:29:11.638834 (+ 292us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:29:11.639069 (+ 235us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:29:11.640104 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:29:11.643452 (+ 3348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:29:11.646384 (+ 2932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:29:11.647868 (+ 1484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:29:11.649881 (+ 2013us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:29:11.650147 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0810 03:29:11.651041 (+ 894us) server_negotiation.cc:299] Negotiation successful
0810 03:29:11.651245 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11629,"thread_start_us":13525,"threads_started":1}
I20260810 03:29:11.652323 16330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:29:11.623230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.2:43577 (local address 127.0.0.1:52122)
0810 03:29:11.638210 (+ 14980us) negotiation.cc:107] Waiting for socket to connect
0810 03:29:11.638230 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 03:29:11.638370 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:29:11.639427 (+ 1057us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:29:11.639433 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:29:11.639463 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:29:11.639858 (+ 395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:29:11.639870 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:29:11.643652 (+ 3782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:29:11.643659 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:29:11.645992 (+ 2333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:29:11.646104 (+ 112us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:29:11.650308 (+ 4204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:29:11.650316 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:29:11.650501 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:29:11.650538 (+ 37us) client_negotiation.cc:769] Sending connection context
0810 03:29:11.650630 (+ 92us) client_negotiation.cc:240] Negotiation successful
0810 03:29:11.650790 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14834,"thread_start_us":6604,"threads_started":1}
I20260810 03:29:11.656556 15953 tablet_service.cc:1467] Processing CreateTablet for tablet 28c72d92e80c48f2ba97fea2f9024684 (DEFAULT_TABLE table=test-table [id=422f9493af764deb98c1504a0fdc5074]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:29:11.657440 16339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:29:11.631095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.1:38941 (local address 127.0.0.1:57364)
0810 03:29:11.646511 (+ 15416us) negotiation.cc:107] Waiting for socket to connect
0810 03:29:11.646538 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0810 03:29:11.646835 (+ 297us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:29:11.648110 (+ 1275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:29:11.648115 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:29:11.648147 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:29:11.648650 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:29:11.648665 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:29:11.650988 (+ 2323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:29:11.650993 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:29:11.654171 (+ 3178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:29:11.654198 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:29:11.656601 (+ 2403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:29:11.656609 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:29:11.656803 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:29:11.656838 (+ 35us) client_negotiation.cc:769] Sending connection context
0810 03:29:11.656928 (+ 90us) client_negotiation.cc:240] Negotiation successful
0810 03:29:11.657080 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15267,"thread_start_us":15093,"threads_started":1}
I20260810 03:29:11.655297 16177 tablet_service.cc:1467] Processing CreateTablet for tablet 28c72d92e80c48f2ba97fea2f9024684 (DEFAULT_TABLE table=test-table [id=422f9493af764deb98c1504a0fdc5074]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:29:11.658358 16340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:29:11.631326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57364 (local address 127.27.161.1:38941)
0810 03:29:11.637505 (+ 6179us) server_negotiation.cc:206] Beginning negotiation
0810 03:29:11.637514 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 03:29:11.646691 (+ 9177us) server_negotiation.cc:407] Connection header received
0810 03:29:11.647090 (+ 399us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:29:11.647100 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:29:11.647430 (+ 330us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:29:11.647683 (+ 253us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:29:11.648884 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:29:11.650798 (+ 1914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:29:11.654425 (+ 3627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:29:11.655994 (+ 1569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:29:11.656121 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:29:11.656365 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0810 03:29:11.657818 (+ 1453us) server_negotiation.cc:299] Negotiation successful
0810 03:29:11.658038 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5988,"thread_start_us":6407,"threads_started":1}
I20260810 03:29:11.658453 16177 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28c72d92e80c48f2ba97fea2f9024684. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:29:11.658453 15953 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28c72d92e80c48f2ba97fea2f9024684. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:29:11.663110 15708 tablet_service.cc:1467] Processing CreateTablet for tablet 28c72d92e80c48f2ba97fea2f9024684 (DEFAULT_TABLE table=test-table [id=422f9493af764deb98c1504a0fdc5074]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:29:11.664950 15708 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28c72d92e80c48f2ba97fea2f9024684. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:29:11.670667 16061 heartbeater.cc:503] Master 127.27.161.62:46293 was elected leader, sending a full tablet report...
I20260810 03:29:11.682714 16342 tablet_bootstrap.cc:492] T 28c72d92e80c48f2ba97fea2f9024684 P b8be23f09895463696cbe48e639382c1: Bootstrap starting.
I20260810 03:29:11.688634 16342 tablet_bootstrap.cc:654] T 28c72d92e80c48f2ba97fea2f9024684 P b8be23f09895463696cbe48e639382c1: Neither blocks nor log segments found. Creating new log.
I20260810 03:29:11.689308 16343 tablet_bootstrap.cc:492] T 28c72d92e80c48f2ba97fea2f9024684 P d104ff4d980646909d8d759d0d488fc8: Bootstrap starting.
I20260810 03:29:11.690472 16342 log.cc:824] T 28c72d92e80c48f2ba97fea2f9024684 P b8be23f09895463696cbe48e639382c1: Log is configured to *not* fsync() on all Append() calls
I20260810 03:29:11.693657 16343 tablet_bootstrap.cc:654] T 28c72d92e80c48f2ba97fea2f9024684 P d104ff4d980646909d8d759d0d488fc8: Neither blocks nor log segments found. Creating new log.
I20260810 03:29:11.695854 16347 tablet_bootstrap.cc:492] T 28c72d92e80c48f2ba97fea2f9024684 P fc38f4e60c47484abfefa958e7104dcb: Bootstrap starting.
I20260810 03:29:11.699096 16343 log.cc:824] T 28c72d92e80c48f2ba97fea2f9024684 P d104ff4d980646909d8d759d0d488fc8: Log is configured to *not* fsync() on all Append() calls
I20260810 03:29:11.700639 16347 tablet_bootstrap.cc:654] T 28c72d92e80c48f2ba97fea2f9024684 P fc38f4e60c47484abfefa958e7104dcb: Neither blocks nor log segments found. Creating new log.
I20260810 03:29:11.700724 16342 tablet_bootstrap.cc:492] T 28c72d92e80c48f2ba97fea2f9024684 P b8be23f09895463696cbe48e639382c1: No bootstrap required, opened a new log
I20260810 03:29:11.701092 16342 ts_tablet_manager.cc:1397] T 28c72d92e80c48f2ba97fea2f9024684 P b8be23f09895463696cbe48e639382c1: Time spent bootstrapping tablet: real 0.019s user 0.008s sys 0.008s
I20260810 03:29:11.701686 16347 log.cc:824] T 28c72d92e80c48f2ba97fea2f9024684 P fc38f4e60c47484abfefa958e7104dcb: Log is configured to *not* fsync() on all Append() calls
I20260810 03:29:11.712251 16342 raft_consensus.cc:348] T 28c72d92e80c48f2ba97fea2f9024684 P b8be23f09895463696cbe48e639382c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc38f4e60c47484abfefa958e7104dcb" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 42369 } } peers { permanent_uuid: "b8be23f09895463696cbe48e639382c1" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 43577 } } peers { permanent_uuid: "d104ff4d980646909d8d759d0d488fc8" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 38941 } }
I20260810 03:29:11.714454 16342 raft_consensus.cc:374] T 28c72d92e80c48f2ba97fea2f9024684 P b8be23f09895463696cbe48e639382c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:29:11.714728 16342 raft_consensus.cc:729] T 28c72d92e80c48f2ba97fea2f9024684 P b8be23f09895463696cbe48e639382c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8be23f09895463696cbe48e639382c1, State: Initialized, Role: FOLLOWER
I20260810 03:29:11.715826 16342 consensus_queue.cc:260] T 28c72d92e80c48f2ba97fea2f9024684 P b8be23f09895463696cbe48e639382c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc38f4e60c47484abfefa958e7104dcb" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 42369 } } peers { permanent_uuid: "b8be23f09895463696cbe48e639382c1" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 43577 } } peers { permanent_uuid: "d104ff4d980646909d8d759d0d488fc8" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 38941 } }
I20260810 03:29:11.716279 16347 tablet_bootstrap.cc:492] T 28c72d92e80c48f2ba97fea2f9024684 P fc38f4e60c47484abfefa958e7104dcb: No bootstrap required, opened a new log
I20260810 03:29:11.716641 16347 ts_tablet_manager.cc:1397] T 28c72d92e80c48f2ba97fea2f9024684 P fc38f4e60c47484abfefa958e7104dcb: Time spent bootstrapping tablet: real 0.021s user 0.004s sys 0.012s
I20260810 03:29:11.719476 16342 ts_tablet_manager.cc:1428] T 28c72d92e80c48f2ba97fea2f9024684 P b8be23f09895463696cbe48e639382c1: Time spent starting tablet: real 0.018s user 0.013s sys 0.003s
I20260810 03:29:11.722096 16343 tablet_bootstrap.cc:492] T 28c72d92e80c48f2ba97fea2f9024684 P d104ff4d980646909d8d759d0d488fc8: No bootstrap required, opened a new log
I20260810 03:29:11.722424 16343 ts_tablet_manager.cc:1397] T 28c72d92e80c48f2ba97fea2f9024684 P d104ff4d980646909d8d759d0d488fc8: Time spent bootstrapping tablet: real 0.034s user 0.013s sys 0.005s
I20260810 03:29:11.728655 16347 raft_consensus.cc:348] T 28c72d92e80c48f2ba97fea2f9024684 P fc38f4e60c47484abfefa958e7104dcb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc38f4e60c47484abfefa958e7104dcb" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 42369 } } peers { permanent_uuid: "b8be23f09895463696cbe48e639382c1" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 43577 } } peers { permanent_uuid: "d104ff4d980646909d8d759d0d488fc8" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 38941 } }
I20260810 03:29:11.729195 16347 raft_consensus.cc:374] T 28c72d92e80c48f2ba97fea2f9024684 P fc38f4e60c47484abfefa958e7104dcb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:29:11.729313 16347 raft_consensus.cc:729] T 28c72d92e80c48f2ba97fea2f9024684 P fc38f4e60c47484abfefa958e7104dcb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc38f4e60c47484abfefa958e7104dcb, State: Initialized, Role: FOLLOWER
I20260810 03:29:11.730046 16347 consensus_queue.cc:260] T 28c72d92e80c48f2ba97fea2f9024684 P fc38f4e60c47484abfefa958e7104dcb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc38f4e60c47484abfefa958e7104dcb" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 42369 } } peers { permanent_uuid: "b8be23f09895463696cbe48e639382c1" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 43577 } } peers { permanent_uuid: "d104ff4d980646909d8d759d0d488fc8" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 38941 } }
I20260810 03:29:11.731289 16288 heartbeater.cc:503] Master 127.27.161.62:46293 was elected leader, sending a full tablet report...
I20260810 03:29:11.732087 16347 ts_tablet_manager.cc:1428] T 28c72d92e80c48f2ba97fea2f9024684 P fc38f4e60c47484abfefa958e7104dcb: Time spent starting tablet: real 0.015s user 0.016s sys 0.000s
I20260810 03:29:11.732745 16343 raft_consensus.cc:348] T 28c72d92e80c48f2ba97fea2f9024684 P d104ff4d980646909d8d759d0d488fc8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc38f4e60c47484abfefa958e7104dcb" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 42369 } } peers { permanent_uuid: "b8be23f09895463696cbe48e639382c1" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 43577 } } peers { permanent_uuid: "d104ff4d980646909d8d759d0d488fc8" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 38941 } }
I20260810 03:29:11.733143 16343 raft_consensus.cc:374] T 28c72d92e80c48f2ba97fea2f9024684 P d104ff4d980646909d8d759d0d488fc8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:29:11.733239 16343 raft_consensus.cc:729] T 28c72d92e80c48f2ba97fea2f9024684 P d104ff4d980646909d8d759d0d488fc8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d104ff4d980646909d8d759d0d488fc8, State: Initialized, Role: FOLLOWER
I20260810 03:29:11.733868 16343 consensus_queue.cc:260] T 28c72d92e80c48f2ba97fea2f9024684 P d104ff4d980646909d8d759d0d488fc8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc38f4e60c47484abfefa958e7104dcb" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 42369 } } peers { permanent_uuid: "b8be23f09895463696cbe48e639382c1" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 43577 } } peers { permanent_uuid: "d104ff4d980646909d8d759d0d488fc8" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 38941 } }
I20260810 03:29:11.735056 15818 heartbeater.cc:503] Master 127.27.161.62:46293 was elected leader, sending a full tablet report...
I20260810 03:29:11.735841 16343 ts_tablet_manager.cc:1428] T 28c72d92e80c48f2ba97fea2f9024684 P d104ff4d980646909d8d759d0d488fc8: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
I20260810 03:29:11.755213 16352 raft_consensus.cc:482] T 28c72d92e80c48f2ba97fea2f9024684 P b8be23f09895463696cbe48e639382c1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 03:29:11.755620 16352 raft_consensus.cc:504] T 28c72d92e80c48f2ba97fea2f9024684 P b8be23f09895463696cbe48e639382c1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc38f4e60c47484abfefa958e7104dcb" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 42369 } } peers { permanent_uuid: "b8be23f09895463696cbe48e639382c1" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 43577 } } peers { permanent_uuid: "d104ff4d980646909d8d759d0d488fc8" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 38941 } }
I20260810 03:29:11.758142 16352 leader_election.cc:288] T 28c72d92e80c48f2ba97fea2f9024684 P b8be23f09895463696cbe48e639382c1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fc38f4e60c47484abfefa958e7104dcb (127.27.161.3:42369), d104ff4d980646909d8d759d0d488fc8 (127.27.161.1:38941)
W20260810 03:29:11.765524 16290 tablet.cc:2369] T 28c72d92e80c48f2ba97fea2f9024684 P fc38f4e60c47484abfefa958e7104dcb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 03:29:11.767055 16338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:29:11.759118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.161.2:54885 (local address 127.27.161.3:42369)
0810 03:29:11.759334 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0810 03:29:11.759342 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 03:29:11.759881 (+ 539us) server_negotiation.cc:407] Connection header received
0810 03:29:11.760068 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:29:11.760073 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:29:11.760140 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:29:11.760228 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:29:11.760932 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:29:11.761699 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:29:11.764531 (+ 2832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:29:11.765770 (+ 1239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:29:11.765945 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:29:11.766059 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0810 03:29:11.766644 (+ 585us) server_negotiation.cc:299] Negotiation successful
0810 03:29:11.766796 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260810 03:29:11.767807 16340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:29:11.758600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.161.2:55211 (local address 127.27.161.1:38941)
0810 03:29:11.758815 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0810 03:29:11.758821 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:29:11.759475 (+ 654us) server_negotiation.cc:407] Connection header received
0810 03:29:11.759811 (+ 336us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:29:11.759819 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:29:11.759898 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:29:11.759959 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:29:11.760846 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:29:11.761909 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:29:11.764572 (+ 2663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:29:11.766082 (+ 1510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:29:11.766315 (+ 233us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:29:11.766410 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0810 03:29:11.767376 (+ 966us) server_negotiation.cc:299] Negotiation successful
0810 03:29:11.767550 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260810 03:29:11.767838 16358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:29:11.758498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.1:38941 (local address 127.27.161.2:55211)
0810 03:29:11.759351 (+ 853us) negotiation.cc:107] Waiting for socket to connect
0810 03:29:11.759372 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0810 03:29:11.759623 (+ 251us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:29:11.760172 (+ 549us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:29:11.760176 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:29:11.760202 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:29:11.760639 (+ 437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:29:11.760656 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:29:11.762067 (+ 1411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:29:11.762073 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:29:11.764387 (+ 2314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:29:11.764398 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:29:11.766685 (+ 2287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:29:11.766693 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:29:11.766976 (+ 283us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:29:11.767004 (+ 28us) client_negotiation.cc:769] Sending connection context
0810 03:29:11.767104 (+ 100us) client_negotiation.cc:240] Negotiation successful
0810 03:29:11.767265 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":708,"mutex_wait_us":31,"thread_start_us":464,"threads_started":1}
I20260810 03:29:11.767908 16359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:29:11.758548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.3:42369 (local address 127.27.161.2:54885)
0810 03:29:11.759758 (+ 1210us) negotiation.cc:107] Waiting for socket to connect
0810 03:29:11.759776 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 03:29:11.759925 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:29:11.760265 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:29:11.760270 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:29:11.760317 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:29:11.760760 (+ 443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:29:11.760771 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:29:11.761853 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:29:11.761859 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:29:11.764344 (+ 2485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:29:11.764364 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:29:11.766237 (+ 1873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:29:11.766245 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:29:11.766425 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:29:11.766463 (+ 38us) client_negotiation.cc:769] Sending connection context
0810 03:29:11.767336 (+ 873us) client_negotiation.cc:240] Negotiation successful
0810 03:29:11.767493 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1085,"mutex_wait_us":3,"thread_start_us":901,"threads_started":1}
I20260810 03:29:11.769048 16209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28c72d92e80c48f2ba97fea2f9024684" candidate_uuid: "b8be23f09895463696cbe48e639382c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc38f4e60c47484abfefa958e7104dcb" is_pre_election: true
I20260810 03:29:11.769284 15752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28c72d92e80c48f2ba97fea2f9024684" candidate_uuid: "b8be23f09895463696cbe48e639382c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d104ff4d980646909d8d759d0d488fc8" is_pre_election: true
I20260810 03:29:11.769636 16209 raft_consensus.cc:2454] T 28c72d92e80c48f2ba97fea2f9024684 P fc38f4e60c47484abfefa958e7104dcb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b8be23f09895463696cbe48e639382c1 in term 0.
I20260810 03:29:11.770025 15752 raft_consensus.cc:2454] T 28c72d92e80c48f2ba97fea2f9024684 P d104ff4d980646909d8d759d0d488fc8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b8be23f09895463696cbe48e639382c1 in term 0.
I20260810 03:29:11.770440 15881 leader_election.cc:302] T 28c72d92e80c48f2ba97fea2f9024684 P b8be23f09895463696cbe48e639382c1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b8be23f09895463696cbe48e639382c1, fc38f4e60c47484abfefa958e7104dcb; no voters:
I20260810 03:29:11.770957 16352 raft_consensus.cc:2781] T 28c72d92e80c48f2ba97fea2f9024684 P b8be23f09895463696cbe48e639382c1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 03:29:11.771153 16352 raft_consensus.cc:482] T 28c72d92e80c48f2ba97fea2f9024684 P b8be23f09895463696cbe48e639382c1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 03:29:11.771286 16352 raft_consensus.cc:3037] T 28c72d92e80c48f2ba97fea2f9024684 P b8be23f09895463696cbe48e639382c1 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:29:11.774331 16352 raft_consensus.cc:504] T 28c72d92e80c48f2ba97fea2f9024684 P b8be23f09895463696cbe48e639382c1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc38f4e60c47484abfefa958e7104dcb" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 42369 } } peers { permanent_uuid: "b8be23f09895463696cbe48e639382c1" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 43577 } } peers { permanent_uuid: "d104ff4d980646909d8d759d0d488fc8" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 38941 } }
I20260810 03:29:11.775424 16352 leader_election.cc:288] T 28c72d92e80c48f2ba97fea2f9024684 P b8be23f09895463696cbe48e639382c1 [CANDIDATE]: Term 1 election: Requested vote from peers fc38f4e60c47484abfefa958e7104dcb (127.27.161.3:42369), d104ff4d980646909d8d759d0d488fc8 (127.27.161.1:38941)
I20260810 03:29:11.776070 16209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28c72d92e80c48f2ba97fea2f9024684" candidate_uuid: "b8be23f09895463696cbe48e639382c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc38f4e60c47484abfefa958e7104dcb"
I20260810 03:29:11.776110 15752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28c72d92e80c48f2ba97fea2f9024684" candidate_uuid: "b8be23f09895463696cbe48e639382c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d104ff4d980646909d8d759d0d488fc8"
I20260810 03:29:11.776343 16209 raft_consensus.cc:3037] T 28c72d92e80c48f2ba97fea2f9024684 P fc38f4e60c47484abfefa958e7104dcb [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:29:11.776342 15752 raft_consensus.cc:3037] T 28c72d92e80c48f2ba97fea2f9024684 P d104ff4d980646909d8d759d0d488fc8 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:29:11.780169 16209 raft_consensus.cc:2454] T 28c72d92e80c48f2ba97fea2f9024684 P fc38f4e60c47484abfefa958e7104dcb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b8be23f09895463696cbe48e639382c1 in term 1.
I20260810 03:29:11.780169 15752 raft_consensus.cc:2454] T 28c72d92e80c48f2ba97fea2f9024684 P d104ff4d980646909d8d759d0d488fc8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b8be23f09895463696cbe48e639382c1 in term 1.
I20260810 03:29:11.780789 15881 leader_election.cc:302] T 28c72d92e80c48f2ba97fea2f9024684 P b8be23f09895463696cbe48e639382c1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b8be23f09895463696cbe48e639382c1, fc38f4e60c47484abfefa958e7104dcb; no voters:
I20260810 03:29:11.781292 16352 raft_consensus.cc:2781] T 28c72d92e80c48f2ba97fea2f9024684 P b8be23f09895463696cbe48e639382c1 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:29:11.782161 16352 raft_consensus.cc:686] T 28c72d92e80c48f2ba97fea2f9024684 P b8be23f09895463696cbe48e639382c1 [term 1 LEADER]: Becoming Leader. State: Replica: b8be23f09895463696cbe48e639382c1, State: Running, Role: LEADER
I20260810 03:29:11.782765 16352 consensus_queue.cc:237] T 28c72d92e80c48f2ba97fea2f9024684 P b8be23f09895463696cbe48e639382c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc38f4e60c47484abfefa958e7104dcb" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 42369 } } peers { permanent_uuid: "b8be23f09895463696cbe48e639382c1" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 43577 } } peers { permanent_uuid: "d104ff4d980646909d8d759d0d488fc8" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 38941 } }
I20260810 03:29:11.787860 15487 catalog_manager.cc:5495] T 28c72d92e80c48f2ba97fea2f9024684 P b8be23f09895463696cbe48e639382c1 reported cstate change: term changed from 0 to 1, leader changed from <none> to b8be23f09895463696cbe48e639382c1 (127.27.161.2). New cstate: current_term: 1 leader_uuid: "b8be23f09895463696cbe48e639382c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc38f4e60c47484abfefa958e7104dcb" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 42369 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b8be23f09895463696cbe48e639382c1" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 43577 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d104ff4d980646909d8d759d0d488fc8" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 38941 } health_report { overall_health: UNKNOWN } } }
I20260810 03:29:11.835817 16335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:29:11.828187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52130 (local address 127.27.161.2:43577)
0810 03:29:11.828442 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0810 03:29:11.828449 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:29:11.828471 (+ 22us) server_negotiation.cc:407] Connection header received
0810 03:29:11.828601 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:29:11.828606 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:29:11.828726 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:29:11.828904 (+ 178us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 03:29:11.829239 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:29:11.830871 (+ 1632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:29:11.832765 (+ 1894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:29:11.833572 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:29:11.833701 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:29:11.834414 (+ 713us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 03:29:11.835178 (+ 764us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 03:29:11.835340 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0810 03:29:11.835421 (+ 81us) server_negotiation.cc:299] Negotiation successful
0810 03:29:11.835517 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260810 03:29:11.845809 16209 raft_consensus.cc:1261] T 28c72d92e80c48f2ba97fea2f9024684 P fc38f4e60c47484abfefa958e7104dcb [term 1 FOLLOWER]: Refusing update from remote peer b8be23f09895463696cbe48e639382c1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:29:11.845809 15752 raft_consensus.cc:1261] T 28c72d92e80c48f2ba97fea2f9024684 P d104ff4d980646909d8d759d0d488fc8 [term 1 FOLLOWER]: Refusing update from remote peer b8be23f09895463696cbe48e639382c1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:29:11.847311 16352 consensus_queue.cc:1035] T 28c72d92e80c48f2ba97fea2f9024684 P b8be23f09895463696cbe48e639382c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d104ff4d980646909d8d759d0d488fc8" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 38941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:29:11.847692 16360 consensus_queue.cc:1035] T 28c72d92e80c48f2ba97fea2f9024684 P b8be23f09895463696cbe48e639382c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc38f4e60c47484abfefa958e7104dcb" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 42369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:29:11.886067 16369 mvcc.cc:205] Tried to move back new op lower bound from 7316818132348538880 to 7316818132106747904. Current Snapshot: MvccSnapshot[applied={T|T < 7316818132348538880}]
I20260810 03:29:11.898517 15487 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57108:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 03:29:11.898924 15487 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57108:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 03:29:11.905215 15487 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2e5bc018274444ff9da6c09730cf5140: Sending DeleteTablet for 3 replicas of tablet 28c72d92e80c48f2ba97fea2f9024684
I20260810 03:29:11.906816 15708 tablet_service.cc:1514] Processing DeleteTablet for tablet 28c72d92e80c48f2ba97fea2f9024684 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:29:11 UTC) from {username='kudu', principal='kudu/127.27.161.62@KRBTEST.COM'} at 127.0.0.1:57364
I20260810 03:29:11.909680 16177 tablet_service.cc:1514] Processing DeleteTablet for tablet 28c72d92e80c48f2ba97fea2f9024684 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:29:11 UTC) from {username='kudu', principal='kudu/127.27.161.62@KRBTEST.COM'} at 127.0.0.1:43492
I20260810 03:29:11.918874 15953 tablet_service.cc:1514] Processing DeleteTablet for tablet 28c72d92e80c48f2ba97fea2f9024684 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:29:11 UTC) from {username='kudu', principal='kudu/127.27.161.62@KRBTEST.COM'} at 127.0.0.1:52122
I20260810 03:29:11.922727 15706 tablet_service.cc:1514] Processing DeleteTablet for tablet 28c72d92e80c48f2ba97fea2f9024684 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:29:11 UTC) from {username='kudu', principal='kudu/127.27.161.62@KRBTEST.COM'} at 127.0.0.1:57364
I20260810 03:29:11.922979 16382 tablet_replica.cc:330] T 28c72d92e80c48f2ba97fea2f9024684 P b8be23f09895463696cbe48e639382c1: stopping tablet replica
I20260810 03:29:11.923600 16382 raft_consensus.cc:2229] T 28c72d92e80c48f2ba97fea2f9024684 P b8be23f09895463696cbe48e639382c1 [term 1 LEADER]: Raft consensus shutting down.
I20260810 03:29:11.924465 16382 raft_consensus.cc:2258] T 28c72d92e80c48f2ba97fea2f9024684 P b8be23f09895463696cbe48e639382c1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:29:11.925274 16380 tablet_replica.cc:330] T 28c72d92e80c48f2ba97fea2f9024684 P fc38f4e60c47484abfefa958e7104dcb: stopping tablet replica
I20260810 03:29:11.925820 16380 raft_consensus.cc:2229] T 28c72d92e80c48f2ba97fea2f9024684 P fc38f4e60c47484abfefa958e7104dcb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 03:29:11.926458 16380 raft_consensus.cc:2258] T 28c72d92e80c48f2ba97fea2f9024684 P fc38f4e60c47484abfefa958e7104dcb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:29:11.927351 28292 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iBH5/build/asan/bin/kudu pid 15558
I20260810 03:29:11.927577 16379 tablet_replica.cc:330] T 28c72d92e80c48f2ba97fea2f9024684 P d104ff4d980646909d8d759d0d488fc8: stopping tablet replica
I20260810 03:29:11.928021 16379 raft_consensus.cc:2229] T 28c72d92e80c48f2ba97fea2f9024684 P d104ff4d980646909d8d759d0d488fc8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 03:29:11.928829 16379 raft_consensus.cc:2258] T 28c72d92e80c48f2ba97fea2f9024684 P d104ff4d980646909d8d759d0d488fc8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:29:11.931533 16382 ts_tablet_manager.cc:1905] T 28c72d92e80c48f2ba97fea2f9024684 P b8be23f09895463696cbe48e639382c1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:29:11.934904 16379 ts_tablet_manager.cc:1905] T 28c72d92e80c48f2ba97fea2f9024684 P d104ff4d980646909d8d759d0d488fc8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:29:11.941833 16380 ts_tablet_manager.cc:1905] T 28c72d92e80c48f2ba97fea2f9024684 P fc38f4e60c47484abfefa958e7104dcb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:29:11.944312 16379 ts_tablet_manager.cc:1918] T 28c72d92e80c48f2ba97fea2f9024684 P d104ff4d980646909d8d759d0d488fc8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 03:29:11.944546 16379 log.cc:1196] T 28c72d92e80c48f2ba97fea2f9024684 P d104ff4d980646909d8d759d0d488fc8: Deleting WAL directory at /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786332336319544-28292-0/minicluster-data/ts-0/wal/wals/28c72d92e80c48f2ba97fea2f9024684
I20260810 03:29:11.945281 16379 ts_tablet_manager.cc:1939] T 28c72d92e80c48f2ba97fea2f9024684 P d104ff4d980646909d8d759d0d488fc8: Deleting consensus metadata
I20260810 03:29:11.953359 16380 ts_tablet_manager.cc:1918] T 28c72d92e80c48f2ba97fea2f9024684 P fc38f4e60c47484abfefa958e7104dcb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 03:29:11.953635 16380 log.cc:1196] T 28c72d92e80c48f2ba97fea2f9024684 P fc38f4e60c47484abfefa958e7104dcb: Deleting WAL directory at /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786332336319544-28292-0/minicluster-data/ts-2/wal/wals/28c72d92e80c48f2ba97fea2f9024684
I20260810 03:29:11.954319 16380 ts_tablet_manager.cc:1939] T 28c72d92e80c48f2ba97fea2f9024684 P fc38f4e60c47484abfefa958e7104dcb: Deleting consensus metadata
I20260810 03:29:11.955961 16340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:29:11.928809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57380 (local address 127.27.161.1:38941)
0810 03:29:11.928986 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0810 03:29:11.928992 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:29:11.929011 (+ 19us) server_negotiation.cc:407] Connection header received
0810 03:29:11.929074 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:29:11.929079 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:29:11.929188 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:29:11.929305 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:29:11.935081 (+ 5776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:29:11.936233 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:29:11.938220 (+ 1987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:29:11.938949 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:29:11.939075 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:29:11.945806 (+ 6731us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:29:11.945825 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:29:11.945846 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:29:11.945895 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:29:11.950564 (+ 4669us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:29:11.951652 (+ 1088us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:29:11.951662 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:29:11.951665 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:29:11.951842 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:29:11.952324 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:29:11.952329 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:29:11.952333 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:29:11.952845 (+ 512us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:29:11.953319 (+ 474us) server_negotiation.cc:1035] Waiting for connection context
0810 03:29:11.955327 (+ 2008us) server_negotiation.cc:299] Negotiation successful
0810 03:29:11.955587 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260810 03:29:11.957193 15811 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:29:11.957583 15467 catalog_manager.cc:4841] TS fc38f4e60c47484abfefa958e7104dcb (127.27.161.3:42369): tablet 28c72d92e80c48f2ba97fea2f9024684 (table test-table [id=422f9493af764deb98c1504a0fdc5074]) successfully deleted
I20260810 03:29:11.961123 16382 ts_tablet_manager.cc:1918] T 28c72d92e80c48f2ba97fea2f9024684 P b8be23f09895463696cbe48e639382c1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 03:29:11.961427 16382 log.cc:1196] T 28c72d92e80c48f2ba97fea2f9024684 P b8be23f09895463696cbe48e639382c1: Deleting WAL directory at /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786332336319544-28292-0/minicluster-data/ts-1/wal/wals/28c72d92e80c48f2ba97fea2f9024684
I20260810 03:29:11.962147 16382 ts_tablet_manager.cc:1939] T 28c72d92e80c48f2ba97fea2f9024684 P b8be23f09895463696cbe48e639382c1: Deleting consensus metadata
I20260810 03:29:11.964358 15465 catalog_manager.cc:4841] TS b8be23f09895463696cbe48e639382c1 (127.27.161.2:43577): tablet 28c72d92e80c48f2ba97fea2f9024684 (table test-table [id=422f9493af764deb98c1504a0fdc5074]) successfully deleted
I20260810 03:29:12.116969 28292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iBH5/build/asan/bin/kudu with pid 15558
I20260810 03:29:12.117182 15465 catalog_manager.cc:4841] TS d104ff4d980646909d8d759d0d488fc8 (127.27.161.1:38941): tablet 28c72d92e80c48f2ba97fea2f9024684 (table test-table [id=422f9493af764deb98c1504a0fdc5074]) successfully deleted
W20260810 03:29:12.142511 15465 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.161.1:38941 (error 108)
I20260810 03:29:12.143071 28292 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iBH5/build/asan/bin/kudu pid 15831
I20260810 03:29:12.145114 16339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:29:12.144557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.1:38941 (local address 127.0.0.1:57384)
0810 03:29:12.144774 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0810 03:29:12.144918 (+ 144us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.1:38941: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260810 03:29:12.145550 15465 catalog_manager.cc:4584] TS d104ff4d980646909d8d759d0d488fc8 (127.27.161.1:38941): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 28c72d92e80c48f2ba97fea2f9024684: Network error: Client connection negotiation failed: client connection to 127.27.161.1:38941: connect: Connection refused (error 111)
I20260810 03:29:12.158927 16335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:29:12.144245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52146 (local address 127.27.161.2:43577)
0810 03:29:12.144564 (+ 319us) server_negotiation.cc:206] Beginning negotiation
0810 03:29:12.144569 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:29:12.144592 (+ 23us) server_negotiation.cc:407] Connection header received
0810 03:29:12.144668 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:29:12.144672 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:29:12.144759 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:29:12.144856 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:29:12.146559 (+ 1703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:29:12.147423 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:29:12.148862 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:29:12.149636 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:29:12.149763 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:29:12.152459 (+ 2696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:29:12.152476 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:29:12.152494 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:29:12.152551 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:29:12.155905 (+ 3354us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:29:12.156639 (+ 734us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:29:12.156648 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:29:12.156650 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:29:12.156825 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:29:12.157214 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:29:12.157217 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:29:12.157219 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:29:12.157630 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:29:12.157983 (+ 353us) server_negotiation.cc:1035] Waiting for connection context
0810 03:29:12.158307 (+ 324us) server_negotiation.cc:299] Negotiation successful
0810 03:29:12.158565 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":183}
I20260810 03:29:12.159798 16052 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:29:12.199676 16339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:29:12.198792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.1:38941 (local address 127.0.0.1:57386)
0810 03:29:12.199190 (+ 398us) negotiation.cc:107] Waiting for socket to connect
0810 03:29:12.199512 (+ 322us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.1:38941: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154}
I20260810 03:29:12.200794 16339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:29:12.200394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.1:38941 (local address 127.0.0.1:57394)
0810 03:29:12.200588 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0810 03:29:12.200665 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.1:38941: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260810 03:29:12.261556 16339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:29:12.261029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.1:38941 (local address 127.0.0.1:57402)
0810 03:29:12.261263 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0810 03:29:12.261358 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.1:38941: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260810 03:29:12.262662 16339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:29:12.262290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.1:38941 (local address 127.0.0.1:57408)
0810 03:29:12.262456 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0810 03:29:12.262543 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.1:38941: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260810 03:29:12.279018 28292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iBH5/build/asan/bin/kudu with pid 15831
I20260810 03:29:12.307287 28292 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iBH5/build/asan/bin/kudu pid 16065
I20260810 03:29:12.322703 16386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:29:12.308049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43494 (local address 127.27.161.3:42369)
0810 03:29:12.308522 (+ 473us) server_negotiation.cc:206] Beginning negotiation
0810 03:29:12.308526 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:29:12.308543 (+ 17us) server_negotiation.cc:407] Connection header received
0810 03:29:12.308602 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:29:12.308606 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:29:12.308682 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:29:12.308775 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:29:12.310403 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:29:12.311157 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:29:12.312511 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:29:12.312949 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:29:12.313059 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:29:12.315943 (+ 2884us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:29:12.315972 (+ 29us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:29:12.315999 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:29:12.316064 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:29:12.319575 (+ 3511us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:29:12.320513 (+ 938us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:29:12.320525 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:29:12.320530 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:29:12.320781 (+ 251us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:29:12.321138 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:29:12.321142 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:29:12.321144 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:29:12.321549 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:29:12.321847 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0810 03:29:12.322114 (+ 267us) server_negotiation.cc:299] Negotiation successful
0810 03:29:12.322307 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":378,"thread_start_us":258,"threads_started":1}
I20260810 03:29:12.323426 16283 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:29:12.343688 16339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:29:12.343145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.1:38941 (local address 127.0.0.1:57410)
0810 03:29:12.343374 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0810 03:29:12.343483 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.1:38941: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260810 03:29:12.344802 16339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:29:12.344390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.1:38941 (local address 127.0.0.1:57414)
0810 03:29:12.344595 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0810 03:29:12.344677 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.1:38941: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260810 03:29:12.415200 28292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iBH5/build/asan/bin/kudu with pid 16065
I20260810 03:29:12.442330 28292 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iBH5/build/asan/bin/kudu pid 15442
I20260810 03:29:12.443526 15550 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:29:12.557276 16339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:29:12.556822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.1:38941 (local address 127.0.0.1:57424)
0810 03:29:12.557004 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0810 03:29:12.557086 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.1:38941: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260810 03:29:12.557322 28292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iBH5/build/asan/bin/kudu with pid 15442
I20260810 03:29:12.591226 28292 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260810 03:29:12.938024 28292 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260810 03:29:12.938287 28292 mini_ranger.cc:66] Stopping Ranger...
I20260810 03:29:13.294946 28292 mini_ranger.cc:68] Stopped Ranger
2026-08-10 03:29:13.295 UTC [13957] LOG: received smart shutdown request
2026-08-10 03:29:13.299 UTC [13957] LOG: background worker "logical replication launcher" (PID 13971) exited with exit code 1
2026-08-10 03:29:13.300 UTC [13960] LOG: shutting down
2026-08-10 03:29:13.478 UTC [13957] LOG: database system is shut down
2026-08-10T03:29:13Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (57493 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786332336319544-28292-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 03:29:13 dist-test-slave-vpvm krb5kdc[16403](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 03:29:13 dist-test-slave-vpvm krb5kdc[16403](info): set up 2 sockets
Aug 10 03:29:13 dist-test-slave-vpvm krb5kdc[16403](info): commencing operation
krb5kdc: starting...
W20260810 03:29:14.862136 28292 mini_kdc.cc:121] Time spent starting KDC: real 1.217s user 0.000s sys 0.045s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786332336319544-28292-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786332336319544-28292-0/krb5kdc/test-user.keytab.
Aug 10 03:29:15 dist-test-slave-vpvm krb5kdc[16403](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332555, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T03:29:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T03:29:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.161.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.161.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.161.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.62.keytab.
Entry for principal kudu/127.27.161.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.62.keytab.
I20260810 03:29:15.169762 28292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iBH5/build/asan/bin/kudu
/tmp/dist-test-task24iBH5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786332336319544-28292-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786332336319544-28292-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786332336319544-28292-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786332336319544-28292-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.161.62:33279
--webserver_interface=127.27.161.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.62.keytab
--principal=kudu/127.27.161.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.161.20:42907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.161.62:33279,127.27.161.61:36695,127.27.161.60:37187
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786332336319544-28292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786332336319544-28292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786332336319544-28292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:29:15.527841 16426 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:29:15.528187 16426 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:29:15.528314 16426 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:29:15.536369 16426 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:29:15.536525 16426 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:29:15.536574 16426 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:29:15.536640 16426 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:29:15.536706 16426 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:29:15.547857 16426 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.161.20:42907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786332336319544-28292-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786332336319544-28292-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.161.62:33279,127.27.161.61:36695,127.27.161.60:37187
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.62.keytab
--principal=kudu/127.27.161.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.161.62:33279
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786332336319544-28292-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.161.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786332336319544-28292-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:29:15.549542 16426 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:29:15.551038 16426 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:29:15.564257 16439 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:29:15.564221 16433 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:29:15.564296 16436 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:29:15.565133 16426 server_base.cc:1034] running on GCE node
I20260810 03:29:15.566056 16426 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:29:15.567816 16426 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:29:15.569190 16426 hybrid_clock.cc:648] HybridClock initialized: now 1786332555569116 us; error 71 us; skew 500 ppm
I20260810 03:29:15.569639 16426 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:29:15 dist-test-slave-vpvm krb5kdc[16403](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332555, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.161.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:29:15.573179 16426 init.cc:377] Logged in from keytab as kudu/127.27.161.62@KRBTEST.COM (short username kudu)
I20260810 03:29:15.575604 16426 webserver.cc:460] Webserver started at http://127.27.161.62:36849/ using document root <none> and password file <none>
I20260810 03:29:15.576639 16426 fs_manager.cc:359] Metadata directory not provided
I20260810 03:29:15.576802 16426 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:29:15.577196 16426 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:29:15.581578 16426 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786332336319544-28292-0/minicluster-data/master-0/data/instance:
uuid: "7ac47b87215d4c7baac0e7ff47d65046"
format_stamp: "Formatted at 2026-08-10 03:29:15 on dist-test-slave-vpvm"
server_key: "900ee15e954cd0758ae9be1ab45c8b4a"
server_key_iv: "8de3fd7ca90e8e779e24ce8e93926f64"
server_key_version: "encryptionkey@0"
I20260810 03:29:15.582945 16426 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786332336319544-28292-0/minicluster-data/master-0/wal/instance:
uuid: "7ac47b87215d4c7baac0e7ff47d65046"
format_stamp: "Formatted at 2026-08-10 03:29:15 on dist-test-slave-vpvm"
server_key: "900ee15e954cd0758ae9be1ab45c8b4a"
server_key_iv: "8de3fd7ca90e8e779e24ce8e93926f64"
server_key_version: "encryptionkey@0"
I20260810 03:29:15.589208 16426 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260810 03:29:15.593420 16453 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:29:15.595009 16426 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 03:29:15.595206 16426 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786332336319544-28292-0/minicluster-data/master-0/data,/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786332336319544-28292-0/minicluster-data/master-0/wal
uuid: "7ac47b87215d4c7baac0e7ff47d65046"
format_stamp: "Formatted at 2026-08-10 03:29:15 on dist-test-slave-vpvm"
server_key: "900ee15e954cd0758ae9be1ab45c8b4a"
server_key_iv: "8de3fd7ca90e8e779e24ce8e93926f64"
server_key_version: "encryptionkey@0"
I20260810 03:29:15.595418 16426 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786332336319544-28292-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786332336319544-28292-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786332336319544-28292-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:29:15.612569 16426 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:29:15.617220 16426 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:29:15.617540 16426 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:29:15.637694 16426 rpc_server.cc:307] RPC server started. Bound to: 127.27.161.62:33279
I20260810 03:29:15.637745 16534 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.161.62:33279 every 8 connection(s)
I20260810 03:29:15.639204 16426 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786332336319544-28292-0/minicluster-data/master-0/data/info.pb
I20260810 03:29:15.641310 28292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iBH5/build/asan/bin/kudu as pid 16426
I20260810 03:29:15.641566 28292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786332336319544-28292-0/minicluster-data/master-0/wal/instance
I20260810 03:29:15.642447 28292 external_mini_cluster.cc:1428] Setting key ba24cb74bf66fa5fa0c394309e76a160
I20260810 03:29:15.644202 16535 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:29:15.653847 16535 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.161.62" port: 33279 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 03:29:15 dist-test-slave-vpvm krb5kdc[16403](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332555, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.161.62@KRBTEST.COM for kudu/127.27.161.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.161.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.161.61@KRBTEST.COM" created.
I20260810 03:29:15.678680 16542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:29:15.657703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:33279 (local address 127.0.0.1:57964)
0810 03:29:15.658722 (+ 1019us) negotiation.cc:107] Waiting for socket to connect
0810 03:29:15.658774 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0810 03:29:15.660909 (+ 2135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:29:15.661858 (+ 949us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:29:15.661864 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:29:15.662615 (+ 751us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:29:15.662971 (+ 356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:29:15.662989 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:29:15.664568 (+ 1579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:29:15.664572 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:29:15.665668 (+ 1096us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:29:15.665683 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:29:15.666690 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:29:15.666698 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:29:15.666916 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:29:15.667688 (+ 772us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:29:15.667714 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:29:15.670295 (+ 2581us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:29:15.673985 (+ 3690us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:29:15.674002 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:29:15.674031 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:29:15.674536 (+ 505us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:29:15.674988 (+ 452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:29:15.674994 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:29:15.674998 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:29:15.675162 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:29:15.675998 (+ 836us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:29:15.676007 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:29:15.676255 (+ 248us) client_negotiation.cc:769] Sending connection context
0810 03:29:15.676515 (+ 260us) client_negotiation.cc:240] Negotiation successful
0810 03:29:15.677196 (+ 681us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":706,"mutex_wait_us":175,"thread_start_us":460,"threads_started":1}
I20260810 03:29:15.678716 16544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:29:15.657704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57964 (local address 127.27.161.62:33279)
0810 03:29:15.659642 (+ 1938us) server_negotiation.cc:206] Beginning negotiation
0810 03:29:15.659651 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 03:29:15.659710 (+ 59us) server_negotiation.cc:407] Connection header received
0810 03:29:15.661339 (+ 1629us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:29:15.661347 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:29:15.661609 (+ 262us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:29:15.661805 (+ 196us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:29:15.663173 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:29:15.664412 (+ 1239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:29:15.665810 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:29:15.666444 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:29:15.666634 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:29:15.670596 (+ 3962us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:29:15.670607 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:29:15.670624 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:29:15.670662 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:29:15.673775 (+ 3113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:29:15.674701 (+ 926us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:29:15.674709 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:29:15.674711 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:29:15.674876 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:29:15.675275 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:29:15.675279 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:29:15.675281 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:29:15.675851 (+ 570us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:29:15.676070 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0810 03:29:15.676667 (+ 597us) server_negotiation.cc:299] Negotiation successful
0810 03:29:15.677055 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":77,"server-negotiator.queue_time_us":1707,"spinlock_wait_cycles":29056,"thread_start_us":1484,"threads_started":1}
I20260810 03:29:15.682864 16535 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.161.61" port: 36695 } has no permanent_uuid. Determining permanent_uuid...
I20260810 03:29:15.684324 16542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:29:15.683833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.61:36695 (local address 127.0.0.1:49954)
0810 03:29:15.684045 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0810 03:29:15.684179 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.61:36695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
W20260810 03:29:15.684710 16459 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.161.61:36695: connect: Connection refused (error 111)
I20260810 03:29:15.686431 16542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:29:15.686052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.61:36695 (local address 127.0.0.1:49968)
0810 03:29:15.686202 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0810 03:29:15.686281 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.61:36695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260810 03:29:15.686681 16535 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.161.61:36695: Network error: Client connection negotiation failed: client connection to 127.27.161.61:36695: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.161.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.61.keytab.
Entry for principal kudu/127.27.161.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.61.keytab.
I20260810 03:29:15.709614 28292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iBH5/build/asan/bin/kudu
/tmp/dist-test-task24iBH5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786332336319544-28292-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786332336319544-28292-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786332336319544-28292-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786332336319544-28292-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.161.61:36695
--webserver_interface=127.27.161.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.61.keytab
--principal=kudu/127.27.161.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.161.20:42907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.161.62:33279,127.27.161.61:36695,127.27.161.60:37187
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786332336319544-28292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786332336319544-28292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786332336319544-28292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 03:29:15.736121 16535 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.161.61" port: 36695 } attempt: 1
I20260810 03:29:15.737396 16542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:29:15.736918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.61:36695 (local address 127.0.0.1:49974)
0810 03:29:15.737120 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0810 03:29:15.737246 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.61:36695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260810 03:29:15.738505 16542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:29:15.738138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.61:36695 (local address 127.0.0.1:49990)
0810 03:29:15.738274 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0810 03:29:15.738356 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.61:36695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260810 03:29:15.738714 16535 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.161.61:36695: Network error: Client connection negotiation failed: client connection to 127.27.161.61:36695: connect: Connection refused (error 111)
I20260810 03:29:15.806988 16535 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.161.61" port: 36695 } attempt: 2
I20260810 03:29:15.808343 16542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:29:15.807708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.61:36695 (local address 127.0.0.1:49998)
0810 03:29:15.808015 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0810 03:29:15.808144 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.61:36695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260810 03:29:15.809492 16542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:29:15.809048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.61:36695 (local address 127.0.0.1:50008)
0810 03:29:15.809217 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0810 03:29:15.809325 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.61:36695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260810 03:29:15.809721 16535 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.161.61:36695: Network error: Client connection negotiation failed: client connection to 127.27.161.61:36695: connect: Connection refused (error 111)
I20260810 03:29:15.901000 16535 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.161.61" port: 36695 } attempt: 3
I20260810 03:29:15.902333 16542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:29:15.901757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.61:36695 (local address 127.0.0.1:50010)
0810 03:29:15.902016 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0810 03:29:15.902156 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.61:36695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260810 03:29:15.903473 16542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:29:15.903008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.61:36695 (local address 127.0.0.1:50024)
0810 03:29:15.903184 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0810 03:29:15.903290 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.61:36695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260810 03:29:15.903726 16535 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.161.61:36695: Network error: Client connection negotiation failed: client connection to 127.27.161.61:36695: connect: Connection refused (error 111)
I20260810 03:29:16.047000 16535 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.161.61" port: 36695 } attempt: 4
I20260810 03:29:16.048322 16542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:29:16.047619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.61:36695 (local address 127.0.0.1:50030)
0810 03:29:16.048011 (+ 392us) negotiation.cc:107] Waiting for socket to connect
0810 03:29:16.048127 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.61:36695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200}
I20260810 03:29:16.049594 16542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:29:16.049126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.61:36695 (local address 127.0.0.1:50040)
0810 03:29:16.049294 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0810 03:29:16.049423 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.61:36695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260810 03:29:16.049862 16535 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.161.61:36695: Network error: Client connection negotiation failed: client connection to 127.27.161.61:36695: connect: Connection refused (error 111)
W20260810 03:29:16.071568 16551 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:29:16.071899 16551 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:29:16.071998 16551 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:29:16.080018 16551 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:29:16.080165 16551 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:29:16.080214 16551 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:29:16.080272 16551 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:29:16.080315 16551 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:29:16.090895 16551 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.161.20:42907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786332336319544-28292-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786332336319544-28292-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.27.161.62:33279,127.27.161.61:36695,127.27.161.60:37187
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.61.keytab
--principal=kudu/127.27.161.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.161.61:36695
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786332336319544-28292-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.27.161.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786332336319544-28292-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:29:16.092324 16551 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:29:16.093833 16551 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:29:16.106506 16558 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:29:16.106484 16561 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:29:16.106443 16563 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:29:16.107586 16551 server_base.cc:1034] running on GCE node
I20260810 03:29:16.108445 16551 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:29:16.110009 16551 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:29:16.111214 16551 hybrid_clock.cc:648] HybridClock initialized: now 1786332556111158 us; error 59 us; skew 500 ppm
I20260810 03:29:16.111567 16551 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:29:16 dist-test-slave-vpvm krb5kdc[16403](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332556, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.161.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:29:16.114866 16551 init.cc:377] Logged in from keytab as kudu/127.27.161.61@KRBTEST.COM (short username kudu)
I20260810 03:29:16.116575 16551 webserver.cc:460] Webserver started at http://127.27.161.61:39553/ using document root <none> and password file <none>
I20260810 03:29:16.117334 16551 fs_manager.cc:359] Metadata directory not provided
I20260810 03:29:16.117458 16551 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:29:16.117774 16551 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:29:16.120957 16551 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786332336319544-28292-0/minicluster-data/master-1/data/instance:
uuid: "b3ce0846da144afdb79fe945490b7b83"
format_stamp: "Formatted at 2026-08-10 03:29:16 on dist-test-slave-vpvm"
server_key: "ec3daccffc26bfa59e8e365c3b3a259e"
server_key_iv: "4a87c81c27da18cac0a851c60defbafa"
server_key_version: "encryptionkey@0"
I20260810 03:29:16.121805 16551 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786332336319544-28292-0/minicluster-data/master-1/wal/instance:
uuid: "b3ce0846da144afdb79fe945490b7b83"
format_stamp: "Formatted at 2026-08-10 03:29:16 on dist-test-slave-vpvm"
server_key: "ec3daccffc26bfa59e8e365c3b3a259e"
server_key_iv: "4a87c81c27da18cac0a851c60defbafa"
server_key_version: "encryptionkey@0"
I20260810 03:29:16.127441 16551 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260810 03:29:16.131578 16577 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:29:16.133124 16551 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 03:29:16.133312 16551 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786332336319544-28292-0/minicluster-data/master-1/data,/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786332336319544-28292-0/minicluster-data/master-1/wal
uuid: "b3ce0846da144afdb79fe945490b7b83"
format_stamp: "Formatted at 2026-08-10 03:29:16 on dist-test-slave-vpvm"
server_key: "ec3daccffc26bfa59e8e365c3b3a259e"
server_key_iv: "4a87c81c27da18cac0a851c60defbafa"
server_key_version: "encryptionkey@0"
I20260810 03:29:16.133529 16551 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786332336319544-28292-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786332336319544-28292-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786332336319544-28292-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:29:16.144771 16551 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:29:16.149081 16551 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:29:16.149369 16551 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:29:16.169086 16551 rpc_server.cc:307] RPC server started. Bound to: 127.27.161.61:36695
I20260810 03:29:16.169104 16658 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.161.61:36695 every 8 connection(s)
I20260810 03:29:16.170748 16551 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786332336319544-28292-0/minicluster-data/master-1/data/info.pb
I20260810 03:29:16.174736 16659 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:29:16.180516 28292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iBH5/build/asan/bin/kudu as pid 16551
I20260810 03:29:16.180734 28292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786332336319544-28292-0/minicluster-data/master-1/wal/instance
I20260810 03:29:16.181257 28292 external_mini_cluster.cc:1428] Setting key c61786e5d60c958fb4a41c7611100fb4
I20260810 03:29:16.182723 16659 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.161.62" port: 33279 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 03:29:16 dist-test-slave-vpvm krb5kdc[16403](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332556, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.161.61@KRBTEST.COM for kudu/127.27.161.62@KRBTEST.COM
I20260810 03:29:16.209285 16663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:29:16.186275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57976 (local address 127.27.161.62:33279)
0810 03:29:16.186896 (+ 621us) server_negotiation.cc:206] Beginning negotiation
0810 03:29:16.186904 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 03:29:16.188131 (+ 1227us) server_negotiation.cc:407] Connection header received
0810 03:29:16.190843 (+ 2712us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:29:16.190848 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:29:16.190928 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:29:16.191038 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:29:16.193526 (+ 2488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:29:16.194429 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:29:16.196385 (+ 1956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:29:16.196838 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:29:16.196948 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:29:16.201791 (+ 4843us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:29:16.201797 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:29:16.201800 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:29:16.201854 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:29:16.205181 (+ 3327us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:29:16.206458 (+ 1277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:29:16.206464 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:29:16.206467 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:29:16.206630 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:29:16.207170 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:29:16.207174 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:29:16.207176 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:29:16.207490 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:29:16.207612 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0810 03:29:16.208765 (+ 1153us) server_negotiation.cc:299] Negotiation successful
0810 03:29:16.208984 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":476,"thread_start_us":268,"threads_started":1}
I20260810 03:29:16.210848 16665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:29:16.186997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:33279 (local address 127.0.0.1:57976)
0810 03:29:16.187899 (+ 902us) negotiation.cc:107] Waiting for socket to connect
0810 03:29:16.187963 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0810 03:29:16.190421 (+ 2458us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:29:16.191365 (+ 944us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:29:16.191379 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:29:16.192321 (+ 942us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:29:16.193209 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:29:16.193240 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:29:16.194671 (+ 1431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:29:16.194677 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:29:16.196175 (+ 1498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:29:16.196193 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:29:16.197113 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:29:16.197121 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:29:16.197435 (+ 314us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:29:16.198292 (+ 857us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:29:16.198329 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:29:16.201475 (+ 3146us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:29:16.205412 (+ 3937us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:29:16.205425 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:29:16.205447 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:29:16.206248 (+ 801us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:29:16.206772 (+ 524us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:29:16.206777 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:29:16.206781 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:29:16.207065 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:29:16.207631 (+ 566us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:29:16.207641 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:29:16.208212 (+ 571us) client_negotiation.cc:769] Sending connection context
0810 03:29:16.208578 (+ 366us) client_negotiation.cc:240] Negotiation successful
0810 03:29:16.209225 (+ 647us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":591,"thread_start_us":349,"threads_started":1}
I20260810 03:29:16.212836 16659 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.161.61" port: 36695 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.27.161.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.161.60@KRBTEST.COM" created.
Aug 10 03:29:16 dist-test-slave-vpvm krb5kdc[16403](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332556, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.161.61@KRBTEST.COM for kudu/127.27.161.61@KRBTEST.COM
I20260810 03:29:16.229100 16665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:29:16.214669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.61:36695 (local address 127.0.0.1:50046)
0810 03:29:16.215090 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0810 03:29:16.215109 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 03:29:16.215230 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:29:16.216258 (+ 1028us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:29:16.216263 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:29:16.216764 (+ 501us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:29:16.216978 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:29:16.216991 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:29:16.218434 (+ 1443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:29:16.218440 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:29:16.219295 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:29:16.219307 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:29:16.220194 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:29:16.220200 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:29:16.220395 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:29:16.220968 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:29:16.220986 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:29:16.223235 (+ 2249us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:29:16.226542 (+ 3307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:29:16.226548 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:29:16.226553 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:29:16.226941 (+ 388us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:29:16.227314 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:29:16.227319 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:29:16.227324 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:29:16.227469 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:29:16.228112 (+ 643us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:29:16.228117 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:29:16.228271 (+ 154us) client_negotiation.cc:769] Sending connection context
0810 03:29:16.228394 (+ 123us) client_negotiation.cc:240] Negotiation successful
0810 03:29:16.228698 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116}
I20260810 03:29:16.229195 16668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:29:16.214968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50046 (local address 127.27.161.61:36695)
0810 03:29:16.215763 (+ 795us) server_negotiation.cc:206] Beginning negotiation
0810 03:29:16.215770 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:29:16.215800 (+ 30us) server_negotiation.cc:407] Connection header received
0810 03:29:16.215854 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:29:16.215861 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:29:16.216070 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:29:16.216216 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:29:16.217152 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:29:16.218297 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:29:16.219434 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:29:16.220041 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:29:16.220166 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:29:16.223441 (+ 3275us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:29:16.223454 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:29:16.223472 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:29:16.223505 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:29:16.226388 (+ 2883us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:29:16.227073 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:29:16.227082 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:29:16.227084 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:29:16.227209 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:29:16.227572 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:29:16.227576 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:29:16.227578 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:29:16.227972 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:29:16.228185 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0810 03:29:16.228580 (+ 395us) server_negotiation.cc:299] Negotiation successful
0810 03:29:16.228826 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":622,"thread_start_us":453,"threads_started":1}
I20260810 03:29:16.232230 16659 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.161.60" port: 37187 } has no permanent_uuid. Determining permanent_uuid...
I20260810 03:29:16.233496 16665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:29:16.233026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.60:37187 (local address 127.0.0.1:52840)
0810 03:29:16.233198 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0810 03:29:16.233326 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.60:37187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260810 03:29:16.233788 16580 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.161.60:37187: connect: Connection refused (error 111)
I20260810 03:29:16.235635 16665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:29:16.235136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.60:37187 (local address 127.0.0.1:52846)
0810 03:29:16.235365 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0810 03:29:16.235476 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.60:37187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260810 03:29:16.236198 16659 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.161.60:37187: Network error: Client connection negotiation failed: client connection to 127.27.161.60:37187: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.161.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.60.keytab.
Entry for principal kudu/127.27.161.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.60.keytab.
I20260810 03:29:16.253576 28292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iBH5/build/asan/bin/kudu
/tmp/dist-test-task24iBH5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786332336319544-28292-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786332336319544-28292-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786332336319544-28292-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786332336319544-28292-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.161.60:37187
--webserver_interface=127.27.161.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.60.keytab
--principal=kudu/127.27.161.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.161.20:42907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.161.62:33279,127.27.161.61:36695,127.27.161.60:37187
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786332336319544-28292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786332336319544-28292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786332336319544-28292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 03:29:16.285562 16659 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.161.60" port: 37187 } attempt: 1
I20260810 03:29:16.286808 16665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:29:16.286349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.60:37187 (local address 127.0.0.1:52858)
0810 03:29:16.286540 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0810 03:29:16.286653 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.60:37187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260810 03:29:16.287756 16665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:29:16.287411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.60:37187 (local address 127.0.0.1:52860)
0810 03:29:16.287557 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0810 03:29:16.287643 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.60:37187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260810 03:29:16.287953 16659 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.161.60:37187: Network error: Client connection negotiation failed: client connection to 127.27.161.60:37187: connect: Connection refused (error 111)
I20260810 03:29:16.349200 16535 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.161.61" port: 36695 } attempt: 5
I20260810 03:29:16.356176 16659 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.161.60" port: 37187 } attempt: 2
I20260810 03:29:16.357371 16665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:29:16.356824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.60:37187 (local address 127.0.0.1:52864)
0810 03:29:16.357042 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0810 03:29:16.357159 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.60:37187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
Aug 10 03:29:16 dist-test-slave-vpvm krb5kdc[16403](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332555, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.161.62@KRBTEST.COM for kudu/127.27.161.61@KRBTEST.COM
I20260810 03:29:16.358736 16665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:29:16.358250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.60:37187 (local address 127.0.0.1:52870)
0810 03:29:16.358437 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0810 03:29:16.358554 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.60:37187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260810 03:29:16.359030 16659 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.161.60:37187: Network error: Client connection negotiation failed: client connection to 127.27.161.60:37187: connect: Connection refused (error 111)
I20260810 03:29:16.364338 16668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:29:16.350249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50058 (local address 127.27.161.61:36695)
0810 03:29:16.350514 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0810 03:29:16.350520 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:29:16.350606 (+ 86us) server_negotiation.cc:407] Connection header received
0810 03:29:16.350752 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:29:16.350757 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:29:16.350845 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:29:16.350935 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:29:16.352244 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:29:16.353218 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:29:16.354621 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:29:16.355114 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:29:16.355231 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:29:16.358692 (+ 3461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:29:16.358697 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:29:16.358701 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:29:16.358737 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:29:16.361558 (+ 2821us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:29:16.362373 (+ 815us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:29:16.362380 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:29:16.362383 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:29:16.362519 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:29:16.362907 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:29:16.362911 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:29:16.362913 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:29:16.363239 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:29:16.363380 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0810 03:29:16.363788 (+ 408us) server_negotiation.cc:299] Negotiation successful
0810 03:29:16.364013 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260810 03:29:16.364353 16542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:29:16.350069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.61:36695 (local address 127.0.0.1:50058)
0810 03:29:16.350304 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0810 03:29:16.350328 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0810 03:29:16.350592 (+ 264us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:29:16.350982 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:29:16.350987 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:29:16.351409 (+ 422us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:29:16.351901 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:29:16.351923 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:29:16.353423 (+ 1500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:29:16.353429 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:29:16.354443 (+ 1014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:29:16.354459 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:29:16.355269 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:29:16.355276 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:29:16.355451 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:29:16.356043 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:29:16.356062 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:29:16.358500 (+ 2438us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:29:16.361758 (+ 3258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:29:16.361765 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:29:16.361769 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:29:16.362235 (+ 466us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:29:16.362628 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:29:16.362634 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:29:16.362638 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:29:16.362789 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:29:16.363359 (+ 570us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:29:16.363364 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:29:16.363523 (+ 159us) client_negotiation.cc:769] Sending connection context
0810 03:29:16.363648 (+ 125us) client_negotiation.cc:240] Negotiation successful
0810 03:29:16.363993 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":82}
I20260810 03:29:16.365541 16535 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.161.60" port: 37187 } has no permanent_uuid. Determining permanent_uuid...
I20260810 03:29:16.366890 16542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:29:16.366493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.60:37187 (local address 127.0.0.1:52886)
0810 03:29:16.366666 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0810 03:29:16.366745 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.60:37187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260810 03:29:16.367897 16542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:29:16.367573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.60:37187 (local address 127.0.0.1:52898)
0810 03:29:16.367716 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0810 03:29:16.367795 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.60:37187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260810 03:29:16.368093 16535 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.161.60:37187: Network error: Client connection negotiation failed: client connection to 127.27.161.60:37187: connect: Connection refused (error 111)
I20260810 03:29:16.419340 16535 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.161.60" port: 37187 } attempt: 1
I20260810 03:29:16.420576 16542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:29:16.420111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.60:37187 (local address 127.0.0.1:52904)
0810 03:29:16.420308 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0810 03:29:16.420409 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.60:37187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260810 03:29:16.421602 16542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:29:16.421273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.60:37187 (local address 127.0.0.1:52920)
0810 03:29:16.421422 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0810 03:29:16.421492 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.60:37187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260810 03:29:16.421813 16535 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.161.60:37187: Network error: Client connection negotiation failed: client connection to 127.27.161.60:37187: connect: Connection refused (error 111)
I20260810 03:29:16.450289 16659 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.161.60" port: 37187 } attempt: 3
I20260810 03:29:16.451537 16665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:29:16.451048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.60:37187 (local address 127.0.0.1:52926)
0810 03:29:16.451275 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0810 03:29:16.451365 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.60:37187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260810 03:29:16.452641 16665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:29:16.452226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.60:37187 (local address 127.0.0.1:52942)
0810 03:29:16.452452 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0810 03:29:16.452529 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.60:37187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131}
W20260810 03:29:16.452842 16659 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.161.60:37187: Network error: Client connection negotiation failed: client connection to 127.27.161.60:37187: connect: Connection refused (error 111)
I20260810 03:29:16.490041 16535 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.161.60" port: 37187 } attempt: 2
I20260810 03:29:16.491380 16542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:29:16.490856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.60:37187 (local address 127.0.0.1:52958)
0810 03:29:16.491099 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0810 03:29:16.491200 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.60:37187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260810 03:29:16.492471 16542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:29:16.492116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.60:37187 (local address 127.0.0.1:52968)
0810 03:29:16.492280 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0810 03:29:16.492353 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.60:37187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260810 03:29:16.492688 16535 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.161.60:37187: Network error: Client connection negotiation failed: client connection to 127.27.161.60:37187: connect: Connection refused (error 111)
I20260810 03:29:16.596092 16659 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.161.60" port: 37187 } attempt: 4
I20260810 03:29:16.597450 16665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:29:16.596898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.60:37187 (local address 127.0.0.1:52982)
0810 03:29:16.597188 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0810 03:29:16.597286 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.60:37187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
I20260810 03:29:16.598587 16665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:29:16.598236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.60:37187 (local address 127.0.0.1:52998)
0810 03:29:16.598394 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0810 03:29:16.598468 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.60:37187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260810 03:29:16.598804 16659 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.161.60:37187: Network error: Client connection negotiation failed: client connection to 127.27.161.60:37187: connect: Connection refused (error 111)
I20260810 03:29:16.598881 16535 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.161.60" port: 37187 } attempt: 3
I20260810 03:29:16.599804 16542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:29:16.599399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.60:37187 (local address 127.0.0.1:53008)
0810 03:29:16.599564 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0810 03:29:16.599653 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.60:37187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260810 03:29:16.600862 16542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:29:16.600461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.60:37187 (local address 127.0.0.1:53010)
0810 03:29:16.600684 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0810 03:29:16.600760 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.60:37187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
W20260810 03:29:16.601047 16535 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.161.60:37187: Network error: Client connection negotiation failed: client connection to 127.27.161.60:37187: connect: Connection refused (error 111)
W20260810 03:29:16.628273 16671 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:29:16.628646 16671 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:29:16.628787 16671 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:29:16.636507 16671 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:29:16.636659 16671 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:29:16.636740 16671 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:29:16.636847 16671 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:29:16.636950 16671 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:29:16.647753 16671 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.161.20:42907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786332336319544-28292-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786332336319544-28292-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.27.161.62:33279,127.27.161.61:36695,127.27.161.60:37187
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.60.keytab
--principal=kudu/127.27.161.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.161.60:37187
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786332336319544-28292-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.27.161.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786332336319544-28292-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:29:16.650166 16671 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:29:16.651798 16671 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:29:16.664608 16680 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:29:16.664728 16684 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:29:16.664728 16681 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:29:16.665781 16671 server_base.cc:1034] running on GCE node
I20260810 03:29:16.666697 16671 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:29:16.668264 16671 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:29:16.669495 16671 hybrid_clock.cc:648] HybridClock initialized: now 1786332556669441 us; error 62 us; skew 500 ppm
I20260810 03:29:16.669865 16671 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:29:16 dist-test-slave-vpvm krb5kdc[16403](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332556, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.161.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:29:16.673066 16671 init.cc:377] Logged in from keytab as kudu/127.27.161.60@KRBTEST.COM (short username kudu)
I20260810 03:29:16.674803 16671 webserver.cc:460] Webserver started at http://127.27.161.60:44591/ using document root <none> and password file <none>
I20260810 03:29:16.675626 16671 fs_manager.cc:359] Metadata directory not provided
I20260810 03:29:16.675760 16671 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:29:16.676092 16671 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:29:16.679639 16671 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786332336319544-28292-0/minicluster-data/master-2/data/instance:
uuid: "dcd3c946b96747ab9e081805f862cb8e"
format_stamp: "Formatted at 2026-08-10 03:29:16 on dist-test-slave-vpvm"
server_key: "489c42370517d54e236e4a20e1875485"
server_key_iv: "019152962a8e91744b1c359034e68c32"
server_key_version: "encryptionkey@0"
I20260810 03:29:16.680531 16671 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786332336319544-28292-0/minicluster-data/master-2/wal/instance:
uuid: "dcd3c946b96747ab9e081805f862cb8e"
format_stamp: "Formatted at 2026-08-10 03:29:16 on dist-test-slave-vpvm"
server_key: "489c42370517d54e236e4a20e1875485"
server_key_iv: "019152962a8e91744b1c359034e68c32"
server_key_version: "encryptionkey@0"
I20260810 03:29:16.686390 16671 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260810 03:29:16.690248 16695 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:29:16.691713 16671 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 03:29:16.691898 16671 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786332336319544-28292-0/minicluster-data/master-2/data,/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786332336319544-28292-0/minicluster-data/master-2/wal
uuid: "dcd3c946b96747ab9e081805f862cb8e"
format_stamp: "Formatted at 2026-08-10 03:29:16 on dist-test-slave-vpvm"
server_key: "489c42370517d54e236e4a20e1875485"
server_key_iv: "019152962a8e91744b1c359034e68c32"
server_key_version: "encryptionkey@0"
I20260810 03:29:16.692101 16671 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786332336319544-28292-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786332336319544-28292-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786332336319544-28292-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:29:16.704908 16671 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:29:16.709300 16671 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:29:16.709642 16671 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:29:16.728799 16671 rpc_server.cc:307] RPC server started. Bound to: 127.27.161.60:37187
I20260810 03:29:16.728832 16784 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.161.60:37187 every 8 connection(s)
I20260810 03:29:16.730370 16671 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786332336319544-28292-0/minicluster-data/master-2/data/info.pb
I20260810 03:29:16.734290 16785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:29:16.735757 28292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iBH5/build/asan/bin/kudu as pid 16671
I20260810 03:29:16.735970 28292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786332336319544-28292-0/minicluster-data/master-2/wal/instance
I20260810 03:29:16.736582 28292 external_mini_cluster.cc:1428] Setting key 62b6681d2f3dff640944600acbad7eaf
I20260810 03:29:16.745168 16785 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.161.62" port: 33279 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 03:29:16 dist-test-slave-vpvm krb5kdc[16403](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332555, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.161.60@KRBTEST.COM
Aug 10 03:29:16 dist-test-slave-vpvm krb5kdc[16403](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332556, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.161.60@KRBTEST.COM for kudu/127.27.161.62@KRBTEST.COM
I20260810 03:29:16.762223 16789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:29:16.738648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53024 (local address 127.27.161.60:37187)
0810 03:29:16.739593 (+ 945us) server_negotiation.cc:206] Beginning negotiation
0810 03:29:16.739610 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0810 03:29:16.739663 (+ 53us) server_negotiation.cc:407] Connection header received
0810 03:29:16.741603 (+ 1940us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:29:16.741636 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:29:16.742244 (+ 608us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:29:16.742781 (+ 537us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:29:16.744316 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:29:16.746417 (+ 2101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:29:16.747940 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:29:16.748970 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:29:16.749191 (+ 221us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:29:16.753978 (+ 4787us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:29:16.753990 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:29:16.754007 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:29:16.754058 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:29:16.757897 (+ 3839us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:29:16.758835 (+ 938us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:29:16.758842 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:29:16.758850 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:29:16.759018 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:29:16.759421 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:29:16.759425 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:29:16.759427 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:29:16.760022 (+ 595us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:29:16.760310 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0810 03:29:16.760732 (+ 422us) server_negotiation.cc:299] Negotiation successful
0810 03:29:16.761075 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":646,"thread_start_us":361,"threads_started":1}
I20260810 03:29:16.765100 16790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:29:16.748383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57982 (local address 127.27.161.62:33279)
0810 03:29:16.748989 (+ 606us) server_negotiation.cc:206] Beginning negotiation
0810 03:29:16.748996 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:29:16.749536 (+ 540us) server_negotiation.cc:407] Connection header received
0810 03:29:16.749727 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:29:16.749732 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:29:16.749816 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:29:16.749920 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:29:16.751515 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:29:16.752231 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:29:16.753786 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:29:16.754280 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:29:16.754390 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:29:16.758417 (+ 4027us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:29:16.758426 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:29:16.758430 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:29:16.758479 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:29:16.762105 (+ 3626us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:29:16.762954 (+ 849us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:29:16.762959 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:29:16.762963 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:29:16.763137 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:29:16.763555 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:29:16.763560 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:29:16.763563 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:29:16.763872 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:29:16.763990 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0810 03:29:16.764506 (+ 516us) server_negotiation.cc:299] Negotiation successful
0810 03:29:16.764709 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":460,"thread_start_us":283,"threads_started":1}
I20260810 03:29:16.765154 16792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:29:16.748390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:33279 (local address 127.0.0.1:57982)
0810 03:29:16.749392 (+ 1002us) negotiation.cc:107] Waiting for socket to connect
0810 03:29:16.749442 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0810 03:29:16.749596 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:29:16.750048 (+ 452us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:29:16.750055 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:29:16.750919 (+ 864us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:29:16.751323 (+ 404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:29:16.751338 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:29:16.752470 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:29:16.752474 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:29:16.753611 (+ 1137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:29:16.753625 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:29:16.754488 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:29:16.754495 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:29:16.754704 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:29:16.755302 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:29:16.755327 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:29:16.758184 (+ 2857us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:29:16.762241 (+ 4057us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:29:16.762251 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:29:16.762258 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:29:16.762827 (+ 569us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:29:16.763285 (+ 458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:29:16.763289 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:29:16.763292 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:29:16.763436 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:29:16.764004 (+ 568us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:29:16.764011 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:29:16.764235 (+ 224us) client_negotiation.cc:769] Sending connection context
0810 03:29:16.764428 (+ 193us) client_negotiation.cc:240] Negotiation successful
0810 03:29:16.764797 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":734,"thread_start_us":468,"threads_started":1}
I20260810 03:29:16.766677 16785 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.161.61" port: 36695 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 03:29:16 dist-test-slave-vpvm krb5kdc[16403](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332556, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.161.60@KRBTEST.COM for kudu/127.27.161.61@KRBTEST.COM
Aug 10 03:29:16 dist-test-slave-vpvm krb5kdc[16403](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332555, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.161.62@KRBTEST.COM
I20260810 03:29:16.778258 16535 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.161.60" port: 37187 } attempt: 4
I20260810 03:29:16.781260 16792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:29:16.767528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.61:36695 (local address 127.0.0.1:50066)
0810 03:29:16.767682 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0810 03:29:16.767697 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 03:29:16.767798 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:29:16.768112 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:29:16.768116 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:29:16.768486 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:29:16.768640 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:29:16.768648 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:29:16.769596 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:29:16.769602 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:29:16.770571 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:29:16.770585 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:29:16.771294 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:29:16.771301 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:29:16.771634 (+ 333us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:29:16.772173 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:29:16.772192 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:29:16.774460 (+ 2268us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:29:16.778106 (+ 3646us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:29:16.778111 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:29:16.778114 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:29:16.778595 (+ 481us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:29:16.779426 (+ 831us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:29:16.779431 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:29:16.779435 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:29:16.779632 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:29:16.780163 (+ 531us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:29:16.780168 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:29:16.780353 (+ 185us) client_negotiation.cc:769] Sending connection context
0810 03:29:16.780543 (+ 190us) client_negotiation.cc:240] Negotiation successful
0810 03:29:16.780883 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53,"mutex_wait_us":1}
I20260810 03:29:16.781284 16668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:29:16.767429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50066 (local address 127.27.161.61:36695)
0810 03:29:16.767577 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0810 03:29:16.767583 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:29:16.767755 (+ 172us) server_negotiation.cc:407] Connection header received
0810 03:29:16.767918 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:29:16.767921 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:29:16.767982 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:29:16.768071 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:29:16.768781 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:29:16.769454 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:29:16.770707 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:29:16.771143 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:29:16.771256 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:29:16.774667 (+ 3411us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:29:16.774673 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:29:16.774676 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:29:16.774712 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:29:16.777888 (+ 3176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:29:16.778736 (+ 848us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:29:16.778742 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:29:16.778745 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:29:16.778934 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:29:16.779746 (+ 812us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:29:16.779752 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:29:16.779755 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:29:16.780041 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:29:16.780186 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0810 03:29:16.780670 (+ 484us) server_negotiation.cc:299] Negotiation successful
0810 03:29:16.780920 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260810 03:29:16.782548 16790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:29:16.765848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57986 (local address 127.27.161.62:33279)
0810 03:29:16.766038 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0810 03:29:16.766043 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:29:16.766135 (+ 92us) server_negotiation.cc:407] Connection header received
0810 03:29:16.766309 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:29:16.766313 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:29:16.766387 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:29:16.766465 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:29:16.768000 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:29:16.768714 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:29:16.770092 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:29:16.770503 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:29:16.770616 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:29:16.775435 (+ 4819us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:29:16.775442 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:29:16.775447 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:29:16.775491 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:29:16.778765 (+ 3274us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:29:16.779721 (+ 956us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:29:16.779726 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:29:16.779730 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:29:16.779874 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:29:16.780318 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:29:16.780324 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:29:16.780327 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:29:16.780695 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:29:16.780964 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0810 03:29:16.782130 (+ 1166us) server_negotiation.cc:299] Negotiation successful
0810 03:29:16.782309 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260810 03:29:16.784168 16785 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.161.60" port: 37187 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 03:29:16 dist-test-slave-vpvm krb5kdc[16403](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332555, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.161.62@KRBTEST.COM for kudu/127.27.161.60@KRBTEST.COM
Aug 10 03:29:16 dist-test-slave-vpvm krb5kdc[16403](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332556, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.161.60@KRBTEST.COM for kudu/127.27.161.60@KRBTEST.COM
I20260810 03:29:16.794337 16789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:29:16.779088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53040 (local address 127.27.161.60:37187)
0810 03:29:16.779246 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0810 03:29:16.779250 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:29:16.779266 (+ 16us) server_negotiation.cc:407] Connection header received
0810 03:29:16.779332 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:29:16.779336 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:29:16.779400 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:29:16.779499 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:29:16.780388 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:29:16.781397 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:29:16.782915 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:29:16.783484 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:29:16.783642 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:29:16.787415 (+ 3773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:29:16.787426 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:29:16.787431 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:29:16.787478 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:29:16.790834 (+ 3356us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:29:16.791725 (+ 891us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:29:16.791731 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:29:16.791734 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:29:16.791883 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:29:16.792394 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:29:16.792399 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:29:16.792403 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:29:16.792854 (+ 451us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:29:16.793031 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0810 03:29:16.793543 (+ 512us) server_negotiation.cc:299] Negotiation successful
0810 03:29:16.793781 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260810 03:29:16.794337 16542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:29:16.778910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.60:37187 (local address 127.0.0.1:53040)
0810 03:29:16.779118 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0810 03:29:16.779138 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 03:29:16.779241 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:29:16.779523 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:29:16.779528 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:29:16.780009 (+ 481us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:29:16.780232 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:29:16.780244 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:29:16.781561 (+ 1317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:29:16.781567 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:29:16.782774 (+ 1207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:29:16.782791 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:29:16.783708 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:29:16.783715 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:29:16.783918 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:29:16.784608 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:29:16.784642 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:29:16.787166 (+ 2524us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:29:16.791014 (+ 3848us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:29:16.791020 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:29:16.791024 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:29:16.791564 (+ 540us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:29:16.792020 (+ 456us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:29:16.792026 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:29:16.792030 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:29:16.792251 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:29:16.793037 (+ 786us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:29:16.793043 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:29:16.793241 (+ 198us) client_negotiation.cc:769] Sending connection context
0810 03:29:16.793402 (+ 161us) client_negotiation.cc:240] Negotiation successful
0810 03:29:16.793745 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
Aug 10 03:29:16 dist-test-slave-vpvm krb5kdc[16403](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332555, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.161.61@KRBTEST.COM
I20260810 03:29:16.802253 16792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:29:16.784935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.60:37187 (local address 127.0.0.1:53050)
0810 03:29:16.785115 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0810 03:29:16.785134 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 03:29:16.785233 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:29:16.786373 (+ 1140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:29:16.786378 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:29:16.786874 (+ 496us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:29:16.787095 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:29:16.787109 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:29:16.788415 (+ 1306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:29:16.788421 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:29:16.790020 (+ 1599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:29:16.790038 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:29:16.791027 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:29:16.791034 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:29:16.791273 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:29:16.791863 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:29:16.791892 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:29:16.794848 (+ 2956us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:29:16.799061 (+ 4213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:29:16.799068 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:29:16.799073 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:29:16.799681 (+ 608us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:29:16.800174 (+ 493us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:29:16.800180 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:29:16.800184 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:29:16.800345 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:29:16.800999 (+ 654us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:29:16.801009 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:29:16.801186 (+ 177us) client_negotiation.cc:769] Sending connection context
0810 03:29:16.801336 (+ 150us) client_negotiation.cc:240] Negotiation successful
0810 03:29:16.801756 (+ 420us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55,"mutex_wait_us":1}
I20260810 03:29:16.802258 16535 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ac47b87215d4c7baac0e7ff47d65046: Bootstrap starting.
I20260810 03:29:16.802759 16795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:29:16.785312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53050 (local address 127.27.161.60:37187)
0810 03:29:16.786050 (+ 738us) server_negotiation.cc:206] Beginning negotiation
0810 03:29:16.786055 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:29:16.786077 (+ 22us) server_negotiation.cc:407] Connection header received
0810 03:29:16.786127 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:29:16.786131 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:29:16.786196 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:29:16.786302 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:29:16.787284 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:29:16.788260 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:29:16.790179 (+ 1919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:29:16.790854 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:29:16.790964 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:29:16.795160 (+ 4196us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:29:16.795171 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:29:16.795176 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:29:16.795223 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:29:16.798918 (+ 3695us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:29:16.799818 (+ 900us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:29:16.799825 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:29:16.799829 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:29:16.800049 (+ 220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:29:16.800465 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:29:16.800471 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:29:16.800475 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:29:16.800892 (+ 417us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:29:16.801075 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0810 03:29:16.801464 (+ 389us) server_negotiation.cc:299] Negotiation successful
0810 03:29:16.802156 (+ 692us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":444,"server-negotiator.queue_time_us":578,"thread_start_us":329,"threads_started":1}
I20260810 03:29:16.804807 16668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:29:16.784888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50072 (local address 127.27.161.61:36695)
0810 03:29:16.785073 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0810 03:29:16.785080 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:29:16.785564 (+ 484us) server_negotiation.cc:407] Connection header received
0810 03:29:16.785637 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:29:16.785643 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:29:16.785727 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:29:16.785826 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:29:16.787602 (+ 1776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:29:16.788706 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:29:16.790570 (+ 1864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:29:16.791099 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:29:16.791240 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:29:16.797078 (+ 5838us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:29:16.797086 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:29:16.797090 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:29:16.797136 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:29:16.800778 (+ 3642us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:29:16.801729 (+ 951us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:29:16.801735 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:29:16.801739 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:29:16.801924 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:29:16.802940 (+ 1016us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:29:16.802945 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:29:16.802949 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:29:16.803320 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:29:16.803603 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0810 03:29:16.804162 (+ 559us) server_negotiation.cc:299] Negotiation successful
0810 03:29:16.804420 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260810 03:29:16.806522 16535 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7ac47b87215d4c7baac0e7ff47d65046: Neither blocks nor log segments found. Creating new log.
I20260810 03:29:16.807885 16535 log.cc:824] T 00000000000000000000000000000000 P 7ac47b87215d4c7baac0e7ff47d65046: Log is configured to *not* fsync() on all Append() calls
I20260810 03:29:16.808630 16785 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dcd3c946b96747ab9e081805f862cb8e: Bootstrap starting.
I20260810 03:29:16.811409 16535 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ac47b87215d4c7baac0e7ff47d65046: No bootstrap required, opened a new log
I20260810 03:29:16.812832 16785 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dcd3c946b96747ab9e081805f862cb8e: Neither blocks nor log segments found. Creating new log.
I20260810 03:29:16.814342 16785 log.cc:824] T 00000000000000000000000000000000 P dcd3c946b96747ab9e081805f862cb8e: Log is configured to *not* fsync() on all Append() calls
I20260810 03:29:16.814446 28292 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 03:29:16.819463 16785 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dcd3c946b96747ab9e081805f862cb8e: No bootstrap required, opened a new log
I20260810 03:29:16.824203 16535 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7ac47b87215d4c7baac0e7ff47d65046 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ac47b87215d4c7baac0e7ff47d65046" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 33279 } } peers { permanent_uuid: "b3ce0846da144afdb79fe945490b7b83" member_type: VOTER last_known_addr { host: "127.27.161.61" port: 36695 } } peers { permanent_uuid: "dcd3c946b96747ab9e081805f862cb8e" member_type: VOTER last_known_addr { host: "127.27.161.60" port: 37187 } }
I20260810 03:29:16.824687 16535 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7ac47b87215d4c7baac0e7ff47d65046 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:29:16.824908 16535 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7ac47b87215d4c7baac0e7ff47d65046 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ac47b87215d4c7baac0e7ff47d65046, State: Initialized, Role: FOLLOWER
I20260810 03:29:16.825835 16535 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7ac47b87215d4c7baac0e7ff47d65046 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ac47b87215d4c7baac0e7ff47d65046" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 33279 } } peers { permanent_uuid: "b3ce0846da144afdb79fe945490b7b83" member_type: VOTER last_known_addr { host: "127.27.161.61" port: 36695 } } peers { permanent_uuid: "dcd3c946b96747ab9e081805f862cb8e" member_type: VOTER last_known_addr { host: "127.27.161.60" port: 37187 } }
I20260810 03:29:16.827042 16808 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ac47b87215d4c7baac0e7ff47d65046 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ac47b87215d4c7baac0e7ff47d65046" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 33279 } } peers { permanent_uuid: "b3ce0846da144afdb79fe945490b7b83" member_type: VOTER last_known_addr { host: "127.27.161.61" port: 36695 } } peers { permanent_uuid: "dcd3c946b96747ab9e081805f862cb8e" member_type: VOTER last_known_addr { host: "127.27.161.60" port: 37187 } } }
I20260810 03:29:16.827682 16808 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ac47b87215d4c7baac0e7ff47d65046 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 03:29:16.829005 16535 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7ac47b87215d4c7baac0e7ff47d65046 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:29:16.831003 16785 raft_consensus.cc:348] T 00000000000000000000000000000000 P dcd3c946b96747ab9e081805f862cb8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ac47b87215d4c7baac0e7ff47d65046" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 33279 } } peers { permanent_uuid: "b3ce0846da144afdb79fe945490b7b83" member_type: VOTER last_known_addr { host: "127.27.161.61" port: 36695 } } peers { permanent_uuid: "dcd3c946b96747ab9e081805f862cb8e" member_type: VOTER last_known_addr { host: "127.27.161.60" port: 37187 } }
I20260810 03:29:16.832880 16785 raft_consensus.cc:374] T 00000000000000000000000000000000 P dcd3c946b96747ab9e081805f862cb8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:29:16.833065 16785 raft_consensus.cc:729] T 00000000000000000000000000000000 P dcd3c946b96747ab9e081805f862cb8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dcd3c946b96747ab9e081805f862cb8e, State: Initialized, Role: FOLLOWER
I20260810 03:29:16.834079 16785 consensus_queue.cc:260] T 00000000000000000000000000000000 P dcd3c946b96747ab9e081805f862cb8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ac47b87215d4c7baac0e7ff47d65046" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 33279 } } peers { permanent_uuid: "b3ce0846da144afdb79fe945490b7b83" member_type: VOTER last_known_addr { host: "127.27.161.61" port: 36695 } } peers { permanent_uuid: "dcd3c946b96747ab9e081805f862cb8e" member_type: VOTER last_known_addr { host: "127.27.161.60" port: 37187 } }
I20260810 03:29:16.838668 16814 sys_catalog.cc:455] T 00000000000000000000000000000000 P dcd3c946b96747ab9e081805f862cb8e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ac47b87215d4c7baac0e7ff47d65046" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 33279 } } peers { permanent_uuid: "b3ce0846da144afdb79fe945490b7b83" member_type: VOTER last_known_addr { host: "127.27.161.61" port: 36695 } } peers { permanent_uuid: "dcd3c946b96747ab9e081805f862cb8e" member_type: VOTER last_known_addr { host: "127.27.161.60" port: 37187 } } }
I20260810 03:29:16.841372 16814 sys_catalog.cc:458] T 00000000000000000000000000000000 P dcd3c946b96747ab9e081805f862cb8e [sys.catalog]: This master's current role is: FOLLOWER
I20260810 03:29:16.841204 16785 sys_catalog.cc:564] T 00000000000000000000000000000000 P dcd3c946b96747ab9e081805f862cb8e [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:29:16.861230 16790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:29:16.827978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57988 (local address 127.27.161.62:33279)
0810 03:29:16.828182 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0810 03:29:16.828190 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 03:29:16.831001 (+ 2811us) server_negotiation.cc:407] Connection header received
0810 03:29:16.832405 (+ 1404us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:29:16.832411 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:29:16.832501 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:29:16.832580 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:29:16.836679 (+ 4099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:29:16.838089 (+ 1410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:29:16.840377 (+ 2288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:29:16.841049 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:29:16.841177 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:29:16.850262 (+ 9085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:29:16.850271 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:29:16.850277 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:29:16.850342 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:29:16.855557 (+ 5215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:29:16.856953 (+ 1396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:29:16.856961 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:29:16.856965 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:29:16.857207 (+ 242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:29:16.857749 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:29:16.857756 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:29:16.857759 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:29:16.858235 (+ 476us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:29:16.859695 (+ 1460us) server_negotiation.cc:1035] Waiting for connection context
0810 03:29:16.860416 (+ 721us) server_negotiation.cc:299] Negotiation successful
0810 03:29:16.860743 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260810 03:29:16.864889 16795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:29:16.831843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53052 (local address 127.27.161.60:37187)
0810 03:29:16.832099 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0810 03:29:16.832106 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:29:16.836556 (+ 4450us) server_negotiation.cc:407] Connection header received
0810 03:29:16.836682 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:29:16.836689 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:29:16.836797 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:29:16.836936 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:29:16.840137 (+ 3201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:29:16.841428 (+ 1291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:29:16.844721 (+ 3293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:29:16.845621 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:29:16.845756 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:29:16.850607 (+ 4851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:29:16.850618 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:29:16.850624 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:29:16.850688 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:29:16.858125 (+ 7437us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:29:16.859315 (+ 1190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:29:16.859323 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:29:16.859328 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:29:16.859548 (+ 220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:29:16.860143 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:29:16.860150 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:29:16.860155 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:29:16.860642 (+ 487us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:29:16.860948 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0810 03:29:16.861719 (+ 771us) server_negotiation.cc:299] Negotiation successful
0810 03:29:16.862088 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
W20260810 03:29:16.870735 16461 tablet.cc:2369] T 00000000000000000000000000000000 P 7ac47b87215d4c7baac0e7ff47d65046: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 03:29:16.876950 16668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:29:16.828514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50086 (local address 127.27.161.61:36695)
0810 03:29:16.828694 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0810 03:29:16.828699 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:29:16.835846 (+ 7147us) server_negotiation.cc:407] Connection header received
0810 03:29:16.836024 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:29:16.836030 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:29:16.836135 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:29:16.836236 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:29:16.847607 (+ 11371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:29:16.848880 (+ 1273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:29:16.854915 (+ 6035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:29:16.855677 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:29:16.855794 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:29:16.866912 (+ 11118us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:29:16.866921 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:29:16.866927 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:29:16.867005 (+ 78us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:29:16.871543 (+ 4538us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:29:16.872830 (+ 1287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:29:16.872837 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:29:16.872840 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:29:16.873092 (+ 252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:29:16.873723 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:29:16.873731 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:29:16.873735 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:29:16.874271 (+ 536us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:29:16.874558 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0810 03:29:16.876195 (+ 1637us) server_negotiation.cc:299] Negotiation successful
0810 03:29:16.876520 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
W20260810 03:29:16.883781 16835 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7ac47b87215d4c7baac0e7ff47d65046: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 03:29:16.884043 16835 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 03:29:16.893245 16814 raft_consensus.cc:482] T 00000000000000000000000000000000 P dcd3c946b96747ab9e081805f862cb8e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 03:29:16.893635 16814 raft_consensus.cc:504] T 00000000000000000000000000000000 P dcd3c946b96747ab9e081805f862cb8e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ac47b87215d4c7baac0e7ff47d65046" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 33279 } } peers { permanent_uuid: "b3ce0846da144afdb79fe945490b7b83" member_type: VOTER last_known_addr { host: "127.27.161.61" port: 36695 } } peers { permanent_uuid: "dcd3c946b96747ab9e081805f862cb8e" member_type: VOTER last_known_addr { host: "127.27.161.60" port: 37187 } }
I20260810 03:29:16.882305 16426 master_runner.cc:186] Error getting master registration for 127.27.161.61:36695: OK, instance_id { permanent_uuid: "b3ce0846da144afdb79fe945490b7b83" instance_seqno: 1786332556163422 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260810 03:29:16.894510 16426 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 03:29:16.895705 16814 leader_election.cc:288] T 00000000000000000000000000000000 P dcd3c946b96747ab9e081805f862cb8e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7ac47b87215d4c7baac0e7ff47d65046 (127.27.161.62:33279), b3ce0846da144afdb79fe945490b7b83 (127.27.161.61:36695)
I20260810 03:29:16.898025 16659 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.161.60" port: 37187 } attempt: 5
I20260810 03:29:16.899183 16497 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "dcd3c946b96747ab9e081805f862cb8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ac47b87215d4c7baac0e7ff47d65046" is_pre_election: true
I20260810 03:29:16.899870 16497 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7ac47b87215d4c7baac0e7ff47d65046 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dcd3c946b96747ab9e081805f862cb8e in term 0.
W20260810 03:29:16.901830 16852 catalog_manager.cc:1558] T 00000000000000000000000000000000 P dcd3c946b96747ab9e081805f862cb8e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 03:29:16.902096 16852 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 03:29:16.904808 16697 leader_election.cc:302] T 00000000000000000000000000000000 P dcd3c946b96747ab9e081805f862cb8e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ac47b87215d4c7baac0e7ff47d65046, dcd3c946b96747ab9e081805f862cb8e; no voters:
I20260810 03:29:16.905843 16814 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dcd3c946b96747ab9e081805f862cb8e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 03:29:16.906024 16814 raft_consensus.cc:482] T 00000000000000000000000000000000 P dcd3c946b96747ab9e081805f862cb8e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 03:29:16.906139 16814 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dcd3c946b96747ab9e081805f862cb8e [term 0 FOLLOWER]: Advancing to term 1
Aug 10 03:29:16 dist-test-slave-vpvm krb5kdc[16403](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332556, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.161.61@KRBTEST.COM for kudu/127.27.161.60@KRBTEST.COM
I20260810 03:29:16.913748 16814 raft_consensus.cc:504] T 00000000000000000000000000000000 P dcd3c946b96747ab9e081805f862cb8e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ac47b87215d4c7baac0e7ff47d65046" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 33279 } } peers { permanent_uuid: "b3ce0846da144afdb79fe945490b7b83" member_type: VOTER last_known_addr { host: "127.27.161.61" port: 36695 } } peers { permanent_uuid: "dcd3c946b96747ab9e081805f862cb8e" member_type: VOTER last_known_addr { host: "127.27.161.60" port: 37187 } }
I20260810 03:29:16.916558 16814 leader_election.cc:288] T 00000000000000000000000000000000 P dcd3c946b96747ab9e081805f862cb8e [CANDIDATE]: Term 1 election: Requested vote from peers 7ac47b87215d4c7baac0e7ff47d65046 (127.27.161.62:33279), b3ce0846da144afdb79fe945490b7b83 (127.27.161.61:36695)
I20260810 03:29:16.917115 16497 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "dcd3c946b96747ab9e081805f862cb8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ac47b87215d4c7baac0e7ff47d65046"
I20260810 03:29:16.917392 16497 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7ac47b87215d4c7baac0e7ff47d65046 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:29:16.921619 16497 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7ac47b87215d4c7baac0e7ff47d65046 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dcd3c946b96747ab9e081805f862cb8e in term 1.
I20260810 03:29:16.922343 16697 leader_election.cc:302] T 00000000000000000000000000000000 P dcd3c946b96747ab9e081805f862cb8e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ac47b87215d4c7baac0e7ff47d65046, dcd3c946b96747ab9e081805f862cb8e; no voters:
I20260810 03:29:16.922897 16814 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dcd3c946b96747ab9e081805f862cb8e [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:29:16.925304 16665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:29:16.898600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.60:37187 (local address 127.0.0.1:53066)
0810 03:29:16.898861 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0810 03:29:16.898881 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 03:29:16.898996 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:29:16.902364 (+ 3368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:29:16.902370 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:29:16.902944 (+ 574us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:29:16.903167 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:29:16.903182 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:29:16.904803 (+ 1621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:29:16.904809 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:29:16.906318 (+ 1509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:29:16.906338 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:29:16.907706 (+ 1368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:29:16.907714 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:29:16.907956 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:29:16.910652 (+ 2696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:29:16.910679 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:29:16.914072 (+ 3393us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:29:16.921660 (+ 7588us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:29:16.921667 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:29:16.921672 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:29:16.922385 (+ 713us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:29:16.922961 (+ 576us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:29:16.922981 (+ 20us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:29:16.922985 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:29:16.923192 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:29:16.924119 (+ 927us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:29:16.924126 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:29:16.924325 (+ 199us) client_negotiation.cc:769] Sending connection context
0810 03:29:16.924496 (+ 171us) client_negotiation.cc:240] Negotiation successful
0810 03:29:16.924842 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137}
I20260810 03:29:16.926628 16795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:29:16.900862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53066 (local address 127.27.161.60:37187)
0810 03:29:16.901359 (+ 497us) server_negotiation.cc:206] Beginning negotiation
0810 03:29:16.901365 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:29:16.901384 (+ 19us) server_negotiation.cc:407] Connection header received
0810 03:29:16.901443 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:29:16.901448 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:29:16.901531 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:29:16.901645 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:29:16.903359 (+ 1714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:29:16.904583 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:29:16.906516 (+ 1933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:29:16.907249 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:29:16.907396 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:29:16.914531 (+ 7135us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:29:16.914545 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:29:16.914551 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:29:16.914608 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:29:16.921383 (+ 6775us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:29:16.922584 (+ 1201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:29:16.922592 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:29:16.922596 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:29:16.922784 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:29:16.923379 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:29:16.923385 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:29:16.923389 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:29:16.923816 (+ 427us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:29:16.924036 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0810 03:29:16.925842 (+ 1806us) server_negotiation.cc:299] Negotiation successful
0810 03:29:16.926167 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":356}
I20260810 03:29:16.935637 16659 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b3ce0846da144afdb79fe945490b7b83: Bootstrap starting.
I20260810 03:29:16.910569 16671 master_runner.cc:186] Error getting master registration for 127.27.161.61:36695: OK, instance_id { permanent_uuid: "b3ce0846da144afdb79fe945490b7b83" instance_seqno: 1786332556163422 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260810 03:29:16.940004 16671 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 03:29:16.940666 16814 raft_consensus.cc:686] T 00000000000000000000000000000000 P dcd3c946b96747ab9e081805f862cb8e [term 1 LEADER]: Becoming Leader. State: Replica: dcd3c946b96747ab9e081805f862cb8e, State: Running, Role: LEADER
I20260810 03:29:16.941222 16659 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b3ce0846da144afdb79fe945490b7b83: Neither blocks nor log segments found. Creating new log.
I20260810 03:29:16.941375 16814 consensus_queue.cc:237] T 00000000000000000000000000000000 P dcd3c946b96747ab9e081805f862cb8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ac47b87215d4c7baac0e7ff47d65046" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 33279 } } peers { permanent_uuid: "b3ce0846da144afdb79fe945490b7b83" member_type: VOTER last_known_addr { host: "127.27.161.61" port: 36695 } } peers { permanent_uuid: "dcd3c946b96747ab9e081805f862cb8e" member_type: VOTER last_known_addr { host: "127.27.161.60" port: 37187 } }
I20260810 03:29:16.946287 16659 log.cc:824] T 00000000000000000000000000000000 P b3ce0846da144afdb79fe945490b7b83: Log is configured to *not* fsync() on all Append() calls
I20260810 03:29:16.947885 16854 sys_catalog.cc:455] T 00000000000000000000000000000000 P dcd3c946b96747ab9e081805f862cb8e [sys.catalog]: SysCatalogTable state changed. Reason: New leader dcd3c946b96747ab9e081805f862cb8e. Latest consensus state: current_term: 1 leader_uuid: "dcd3c946b96747ab9e081805f862cb8e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ac47b87215d4c7baac0e7ff47d65046" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 33279 } } peers { permanent_uuid: "b3ce0846da144afdb79fe945490b7b83" member_type: VOTER last_known_addr { host: "127.27.161.61" port: 36695 } } peers { permanent_uuid: "dcd3c946b96747ab9e081805f862cb8e" member_type: VOTER last_known_addr { host: "127.27.161.60" port: 37187 } } }
I20260810 03:29:16.948382 16854 sys_catalog.cc:458] T 00000000000000000000000000000000 P dcd3c946b96747ab9e081805f862cb8e [sys.catalog]: This master's current role is: LEADER
I20260810 03:29:16.957856 16659 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b3ce0846da144afdb79fe945490b7b83: No bootstrap required, opened a new log
I20260810 03:29:16.958638 16858 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 03:29:16.961360 16858 catalog_manager.cc:1484] Initializing Kudu cluster ID...
W20260810 03:29:16.962224 16707 tablet.cc:2369] T 00000000000000000000000000000000 P dcd3c946b96747ab9e081805f862cb8e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 03:29:16.972055 16659 raft_consensus.cc:348] T 00000000000000000000000000000000 P b3ce0846da144afdb79fe945490b7b83 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ac47b87215d4c7baac0e7ff47d65046" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 33279 } } peers { permanent_uuid: "b3ce0846da144afdb79fe945490b7b83" member_type: VOTER last_known_addr { host: "127.27.161.61" port: 36695 } } peers { permanent_uuid: "dcd3c946b96747ab9e081805f862cb8e" member_type: VOTER last_known_addr { host: "127.27.161.60" port: 37187 } }
I20260810 03:29:16.972632 16659 raft_consensus.cc:374] T 00000000000000000000000000000000 P b3ce0846da144afdb79fe945490b7b83 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:29:16.972764 16659 raft_consensus.cc:729] T 00000000000000000000000000000000 P b3ce0846da144afdb79fe945490b7b83 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3ce0846da144afdb79fe945490b7b83, State: Initialized, Role: FOLLOWER
I20260810 03:29:16.973585 16659 consensus_queue.cc:260] T 00000000000000000000000000000000 P b3ce0846da144afdb79fe945490b7b83 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ac47b87215d4c7baac0e7ff47d65046" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 33279 } } peers { permanent_uuid: "b3ce0846da144afdb79fe945490b7b83" member_type: VOTER last_known_addr { host: "127.27.161.61" port: 36695 } } peers { permanent_uuid: "dcd3c946b96747ab9e081805f862cb8e" member_type: VOTER last_known_addr { host: "127.27.161.60" port: 37187 } }
I20260810 03:29:16.978750 16497 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7ac47b87215d4c7baac0e7ff47d65046 [term 1 FOLLOWER]: Refusing update from remote peer dcd3c946b96747ab9e081805f862cb8e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:29:16.980326 16854 consensus_queue.cc:1035] T 00000000000000000000000000000000 P dcd3c946b96747ab9e081805f862cb8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ac47b87215d4c7baac0e7ff47d65046" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 33279 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:29:16.989372 16860 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3ce0846da144afdb79fe945490b7b83 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ac47b87215d4c7baac0e7ff47d65046" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 33279 } } peers { permanent_uuid: "b3ce0846da144afdb79fe945490b7b83" member_type: VOTER last_known_addr { host: "127.27.161.61" port: 36695 } } peers { permanent_uuid: "dcd3c946b96747ab9e081805f862cb8e" member_type: VOTER last_known_addr { host: "127.27.161.60" port: 37187 } } }
I20260810 03:29:16.991539 16612 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b3ce0846da144afdb79fe945490b7b83 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:29:16.991761 16860 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3ce0846da144afdb79fe945490b7b83 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 03:29:16.991050 16659 sys_catalog.cc:564] T 00000000000000000000000000000000 P b3ce0846da144afdb79fe945490b7b83 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:29:16.995993 16617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "dcd3c946b96747ab9e081805f862cb8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3ce0846da144afdb79fe945490b7b83"
I20260810 03:29:16.998026 16618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "dcd3c946b96747ab9e081805f862cb8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3ce0846da144afdb79fe945490b7b83" is_pre_election: true
I20260810 03:29:16.999415 16612 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b3ce0846da144afdb79fe945490b7b83 [term 1 FOLLOWER]: Refusing update from remote peer dcd3c946b96747ab9e081805f862cb8e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:29:17.000378 16854 consensus_queue.cc:1035] T 00000000000000000000000000000000 P dcd3c946b96747ab9e081805f862cb8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3ce0846da144afdb79fe945490b7b83" member_type: VOTER last_known_addr { host: "127.27.161.61" port: 36695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:29:17.017759 16860 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3ce0846da144afdb79fe945490b7b83 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dcd3c946b96747ab9e081805f862cb8e. Latest consensus state: current_term: 1 leader_uuid: "dcd3c946b96747ab9e081805f862cb8e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ac47b87215d4c7baac0e7ff47d65046" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 33279 } } peers { permanent_uuid: "b3ce0846da144afdb79fe945490b7b83" member_type: VOTER last_known_addr { host: "127.27.161.61" port: 36695 } } peers { permanent_uuid: "dcd3c946b96747ab9e081805f862cb8e" member_type: VOTER last_known_addr { host: "127.27.161.60" port: 37187 } } }
I20260810 03:29:17.018298 16860 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3ce0846da144afdb79fe945490b7b83 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 03:29:17.027220 16808 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ac47b87215d4c7baac0e7ff47d65046 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dcd3c946b96747ab9e081805f862cb8e. Latest consensus state: current_term: 1 leader_uuid: "dcd3c946b96747ab9e081805f862cb8e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ac47b87215d4c7baac0e7ff47d65046" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 33279 } } peers { permanent_uuid: "b3ce0846da144afdb79fe945490b7b83" member_type: VOTER last_known_addr { host: "127.27.161.61" port: 36695 } } peers { permanent_uuid: "dcd3c946b96747ab9e081805f862cb8e" member_type: VOTER last_known_addr { host: "127.27.161.60" port: 37187 } } }
I20260810 03:29:17.027594 16808 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ac47b87215d4c7baac0e7ff47d65046 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 03:29:17.027872 16859 mvcc.cc:205] Tried to move back new op lower bound from 7316818153360654336 to 7316818153239199744. Current Snapshot: MvccSnapshot[applied={T|T < 7316818153360654336}]
I20260810 03:29:17.033931 16808 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ac47b87215d4c7baac0e7ff47d65046 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "dcd3c946b96747ab9e081805f862cb8e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ac47b87215d4c7baac0e7ff47d65046" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 33279 } } peers { permanent_uuid: "b3ce0846da144afdb79fe945490b7b83" member_type: VOTER last_known_addr { host: "127.27.161.61" port: 36695 } } peers { permanent_uuid: "dcd3c946b96747ab9e081805f862cb8e" member_type: VOTER last_known_addr { host: "127.27.161.60" port: 37187 } } }
I20260810 03:29:17.034447 16808 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ac47b87215d4c7baac0e7ff47d65046 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 03:29:17.044035 16868 mvcc.cc:205] Tried to move back new op lower bound from 7316818153360654336 to 7316818153239199744. Current Snapshot: MvccSnapshot[applied={T|T < 7316818153360654336}]
I20260810 03:29:17.058241 16814 sys_catalog.cc:455] T 00000000000000000000000000000000 P dcd3c946b96747ab9e081805f862cb8e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "dcd3c946b96747ab9e081805f862cb8e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ac47b87215d4c7baac0e7ff47d65046" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 33279 } } peers { permanent_uuid: "b3ce0846da144afdb79fe945490b7b83" member_type: VOTER last_known_addr { host: "127.27.161.61" port: 36695 } } peers { permanent_uuid: "dcd3c946b96747ab9e081805f862cb8e" member_type: VOTER last_known_addr { host: "127.27.161.60" port: 37187 } } }
I20260810 03:29:17.058686 16814 sys_catalog.cc:458] T 00000000000000000000000000000000 P dcd3c946b96747ab9e081805f862cb8e [sys.catalog]: This master's current role is: LEADER
I20260810 03:29:17.059265 16814 sys_catalog.cc:455] T 00000000000000000000000000000000 P dcd3c946b96747ab9e081805f862cb8e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "dcd3c946b96747ab9e081805f862cb8e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ac47b87215d4c7baac0e7ff47d65046" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 33279 } } peers { permanent_uuid: "b3ce0846da144afdb79fe945490b7b83" member_type: VOTER last_known_addr { host: "127.27.161.61" port: 36695 } } peers { permanent_uuid: "dcd3c946b96747ab9e081805f862cb8e" member_type: VOTER last_known_addr { host: "127.27.161.60" port: 37187 } } }
I20260810 03:29:17.059561 16814 sys_catalog.cc:458] T 00000000000000000000000000000000 P dcd3c946b96747ab9e081805f862cb8e [sys.catalog]: This master's current role is: LEADER
I20260810 03:29:17.061739 16858 catalog_manager.cc:1347] Generated new cluster ID: 386de22b58bb44fab548662e6148c882
I20260810 03:29:17.061904 16858 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 03:29:17.062197 16860 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3ce0846da144afdb79fe945490b7b83 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "dcd3c946b96747ab9e081805f862cb8e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ac47b87215d4c7baac0e7ff47d65046" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 33279 } } peers { permanent_uuid: "b3ce0846da144afdb79fe945490b7b83" member_type: VOTER last_known_addr { host: "127.27.161.61" port: 36695 } } peers { permanent_uuid: "dcd3c946b96747ab9e081805f862cb8e" member_type: VOTER last_known_addr { host: "127.27.161.60" port: 37187 } } }
I20260810 03:29:17.062456 16860 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3ce0846da144afdb79fe945490b7b83 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 03:29:17.079763 16858 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 03:29:17.081890 16858 catalog_manager.cc:1504] Loading token signing keys...
I20260810 03:29:17.108039 16858 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dcd3c946b96747ab9e081805f862cb8e: Generated new TSK 0
I20260810 03:29:17.109242 16858 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 03:29:17.228322 16897 catalog_manager.cc:1259] Loaded cluster ID: 386de22b58bb44fab548662e6148c882
I20260810 03:29:17.228446 16897 catalog_manager.cc:1552] T 00000000000000000000000000000000 P b3ce0846da144afdb79fe945490b7b83: loading cluster ID for follower catalog manager: success
I20260810 03:29:17.232607 16897 catalog_manager.cc:1574] T 00000000000000000000000000000000 P b3ce0846da144afdb79fe945490b7b83: acquiring CA information for follower catalog manager: success
I20260810 03:29:17.235603 16897 catalog_manager.cc:1602] T 00000000000000000000000000000000 P b3ce0846da144afdb79fe945490b7b83: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 03:29:17.886576 16835 catalog_manager.cc:1259] Loaded cluster ID: 386de22b58bb44fab548662e6148c882
I20260810 03:29:17.886678 16835 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 7ac47b87215d4c7baac0e7ff47d65046: loading cluster ID for follower catalog manager: success
I20260810 03:29:17.889134 16835 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 7ac47b87215d4c7baac0e7ff47d65046: acquiring CA information for follower catalog manager: success
I20260810 03:29:17.891047 16835 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 7ac47b87215d4c7baac0e7ff47d65046: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 03:29:18.291097 28292 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iBH5/build/asan/bin/kudu pid 16426
I20260810 03:29:18.292369 16527 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:29:18.399931 28292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iBH5/build/asan/bin/kudu with pid 16426
I20260810 03:29:18.429674 28292 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iBH5/build/asan/bin/kudu pid 16551
I20260810 03:29:18.430898 16653 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:29:18.535339 28292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iBH5/build/asan/bin/kudu with pid 16551
I20260810 03:29:18.563352 28292 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iBH5/build/asan/bin/kudu pid 16671
I20260810 03:29:18.564558 16778 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:29:18.672940 28292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iBH5/build/asan/bin/kudu with pid 16671
2026-08-10T03:29:18Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (5076 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786332336319544-28292-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 03:29:18 dist-test-slave-vpvm krb5kdc[16911](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 03:29:18 dist-test-slave-vpvm krb5kdc[16911](info): set up 2 sockets
Aug 10 03:29:18 dist-test-slave-vpvm krb5kdc[16911](info): commencing operation
krb5kdc: starting...
W20260810 03:29:19.887657 28292 mini_kdc.cc:121] Time spent starting KDC: real 1.167s user 0.004s sys 0.036s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786332336319544-28292-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786332336319544-28292-0/krb5kdc/test-user.keytab.
Aug 10 03:29:20 dist-test-slave-vpvm krb5kdc[16911](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332560, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T03:29:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T03:29:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.161.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.161.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.161.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.62.keytab.
Entry for principal kudu/127.27.161.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.62.keytab.
I20260810 03:29:20.162938 28292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iBH5/build/asan/bin/kudu
/tmp/dist-test-task24iBH5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786332336319544-28292-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786332336319544-28292-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786332336319544-28292-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786332336319544-28292-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.161.62:40973
--webserver_interface=127.27.161.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.62.keytab
--principal=kudu/127.27.161.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.161.20:36747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.161.62:40973
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786332336319544-28292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786332336319544-28292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786332336319544-28292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:29:20.494452 16936 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:29:20.494779 16936 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:29:20.494917 16936 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:29:20.502082 16936 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:29:20.502198 16936 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:29:20.502279 16936 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:29:20.502357 16936 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:29:20.502417 16936 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:29:20.512302 16936 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.161.20:36747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786332336319544-28292-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786332336319544-28292-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.161.62:40973
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.62.keytab
--principal=kudu/127.27.161.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.161.62:40973
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786332336319544-28292-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.161.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786332336319544-28292-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:29:20.513636 16936 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:29:20.514956 16936 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:29:20.526175 16945 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:29:20.526194 16948 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:29:20.526298 16944 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:29:20.527278 16936 server_base.cc:1034] running on GCE node
I20260810 03:29:20.528002 16936 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:29:20.529311 16936 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:29:20.530503 16936 hybrid_clock.cc:648] HybridClock initialized: now 1786332560530445 us; error 43 us; skew 500 ppm
I20260810 03:29:20.530826 16936 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:29:20 dist-test-slave-vpvm krb5kdc[16911](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332560, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.161.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:29:20.536509 16936 init.cc:377] Logged in from keytab as kudu/127.27.161.62@KRBTEST.COM (short username kudu)
I20260810 03:29:20.537918 16936 webserver.cc:460] Webserver started at http://127.27.161.62:38667/ using document root <none> and password file <none>
I20260810 03:29:20.538595 16936 fs_manager.cc:359] Metadata directory not provided
I20260810 03:29:20.538722 16936 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:29:20.539000 16936 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:29:20.541826 16936 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786332336319544-28292-0/minicluster-data/master-0/data/instance:
uuid: "16bc7c23a5714a62a005b8dcf5969876"
format_stamp: "Formatted at 2026-08-10 03:29:20 on dist-test-slave-vpvm"
server_key: "eb6b74c8b8e6b2a23884d58711342e57"
server_key_iv: "7499bee0c0ae37f72dfa71855c75e823"
server_key_version: "encryptionkey@0"
I20260810 03:29:20.542568 16936 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786332336319544-28292-0/minicluster-data/master-0/wal/instance:
uuid: "16bc7c23a5714a62a005b8dcf5969876"
format_stamp: "Formatted at 2026-08-10 03:29:20 on dist-test-slave-vpvm"
server_key: "eb6b74c8b8e6b2a23884d58711342e57"
server_key_iv: "7499bee0c0ae37f72dfa71855c75e823"
server_key_version: "encryptionkey@0"
I20260810 03:29:20.547593 16936 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260810 03:29:20.550961 16961 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:29:20.552232 16936 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260810 03:29:20.552397 16936 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786332336319544-28292-0/minicluster-data/master-0/data,/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786332336319544-28292-0/minicluster-data/master-0/wal
uuid: "16bc7c23a5714a62a005b8dcf5969876"
format_stamp: "Formatted at 2026-08-10 03:29:20 on dist-test-slave-vpvm"
server_key: "eb6b74c8b8e6b2a23884d58711342e57"
server_key_iv: "7499bee0c0ae37f72dfa71855c75e823"
server_key_version: "encryptionkey@0"
I20260810 03:29:20.552583 16936 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786332336319544-28292-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786332336319544-28292-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786332336319544-28292-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:29:20.576190 16936 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:29:20.580031 16936 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:29:20.580294 16936 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:29:20.597702 16936 rpc_server.cc:307] RPC server started. Bound to: 127.27.161.62:40973
I20260810 03:29:20.597733 17053 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.161.62:40973 every 8 connection(s)
I20260810 03:29:20.599009 16936 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786332336319544-28292-0/minicluster-data/master-0/data/info.pb
I20260810 03:29:20.602553 17057 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:29:20.607842 28292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iBH5/build/asan/bin/kudu as pid 16936
I20260810 03:29:20.608076 28292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786332336319544-28292-0/minicluster-data/master-0/wal/instance
I20260810 03:29:20.608899 28292 external_mini_cluster.cc:1428] Setting key c1415ee292cc988812aeffad3b1e047d
I20260810 03:29:20.613574 17057 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16bc7c23a5714a62a005b8dcf5969876: Bootstrap starting.
I20260810 03:29:20.617102 17057 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 16bc7c23a5714a62a005b8dcf5969876: Neither blocks nor log segments found. Creating new log.
I20260810 03:29:20.618099 17057 log.cc:824] T 00000000000000000000000000000000 P 16bc7c23a5714a62a005b8dcf5969876: Log is configured to *not* fsync() on all Append() calls
I20260810 03:29:20.620826 17057 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16bc7c23a5714a62a005b8dcf5969876: No bootstrap required, opened a new log
Aug 10 03:29:20 dist-test-slave-vpvm krb5kdc[16911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332560, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.161.62@KRBTEST.COM
I20260810 03:29:20.629678 17057 raft_consensus.cc:348] T 00000000000000000000000000000000 P 16bc7c23a5714a62a005b8dcf5969876 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16bc7c23a5714a62a005b8dcf5969876" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 40973 } }
I20260810 03:29:20.630174 17057 raft_consensus.cc:374] T 00000000000000000000000000000000 P 16bc7c23a5714a62a005b8dcf5969876 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:29:20.630273 17057 raft_consensus.cc:729] T 00000000000000000000000000000000 P 16bc7c23a5714a62a005b8dcf5969876 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16bc7c23a5714a62a005b8dcf5969876, State: Initialized, Role: FOLLOWER
I20260810 03:29:20.630816 17057 consensus_queue.cc:260] T 00000000000000000000000000000000 P 16bc7c23a5714a62a005b8dcf5969876 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16bc7c23a5714a62a005b8dcf5969876" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 40973 } }
I20260810 03:29:20.631035 17057 raft_consensus.cc:388] T 00000000000000000000000000000000 P 16bc7c23a5714a62a005b8dcf5969876 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 03:29:20.631186 17057 raft_consensus.cc:482] T 00000000000000000000000000000000 P 16bc7c23a5714a62a005b8dcf5969876 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 03:29:20.631251 17061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:29:20.610831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52726 (local address 127.27.161.62:40973)
0810 03:29:20.611641 (+ 810us) server_negotiation.cc:206] Beginning negotiation
0810 03:29:20.611652 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 03:29:20.611698 (+ 46us) server_negotiation.cc:407] Connection header received
0810 03:29:20.613354 (+ 1656us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:29:20.613375 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:29:20.613754 (+ 379us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:29:20.614107 (+ 353us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:29:20.615911 (+ 1804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:29:20.617152 (+ 1241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:29:20.618542 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:29:20.619232 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:29:20.619398 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:29:20.623856 (+ 4458us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:29:20.623864 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:29:20.623878 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:29:20.623915 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:29:20.626962 (+ 3047us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:29:20.627807 (+ 845us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:29:20.627813 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:29:20.627819 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:29:20.628052 (+ 233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:29:20.628449 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:29:20.628453 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:29:20.628455 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:29:20.628991 (+ 536us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:29:20.629203 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0810 03:29:20.629655 (+ 452us) server_negotiation.cc:299] Negotiation successful
0810 03:29:20.630015 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":598,"thread_start_us":374,"threads_started":1}
I20260810 03:29:20.631357 17057 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 16bc7c23a5714a62a005b8dcf5969876 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:29:20.633842 17057 raft_consensus.cc:504] T 00000000000000000000000000000000 P 16bc7c23a5714a62a005b8dcf5969876 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16bc7c23a5714a62a005b8dcf5969876" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 40973 } }
I20260810 03:29:20.634382 17057 leader_election.cc:302] T 00000000000000000000000000000000 P 16bc7c23a5714a62a005b8dcf5969876 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 16bc7c23a5714a62a005b8dcf5969876; no voters:
I20260810 03:29:20.634925 28292 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 03:29:20.634989 17057 leader_election.cc:288] T 00000000000000000000000000000000 P 16bc7c23a5714a62a005b8dcf5969876 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 03:29:20.635099 17064 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 16bc7c23a5714a62a005b8dcf5969876 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:29:20.637166 17057 sys_catalog.cc:564] T 00000000000000000000000000000000 P 16bc7c23a5714a62a005b8dcf5969876 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:29:20.637437 17064 raft_consensus.cc:686] T 00000000000000000000000000000000 P 16bc7c23a5714a62a005b8dcf5969876 [term 1 LEADER]: Becoming Leader. State: Replica: 16bc7c23a5714a62a005b8dcf5969876, State: Running, Role: LEADER
I20260810 03:29:20.638090 17064 consensus_queue.cc:237] T 00000000000000000000000000000000 P 16bc7c23a5714a62a005b8dcf5969876 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16bc7c23a5714a62a005b8dcf5969876" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 40973 } }
I20260810 03:29:20.641151 17066 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16bc7c23a5714a62a005b8dcf5969876 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 16bc7c23a5714a62a005b8dcf5969876. Latest consensus state: current_term: 1 leader_uuid: "16bc7c23a5714a62a005b8dcf5969876" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16bc7c23a5714a62a005b8dcf5969876" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 40973 } } }
I20260810 03:29:20.641172 17065 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16bc7c23a5714a62a005b8dcf5969876 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "16bc7c23a5714a62a005b8dcf5969876" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16bc7c23a5714a62a005b8dcf5969876" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 40973 } } }
I20260810 03:29:20.641590 17065 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16bc7c23a5714a62a005b8dcf5969876 [sys.catalog]: This master's current role is: LEADER
I20260810 03:29:20.641590 17066 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16bc7c23a5714a62a005b8dcf5969876 [sys.catalog]: This master's current role is: LEADER
I20260810 03:29:20.643730 17085 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 03:29:20.647655 17085 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 03:29:20.655916 17085 catalog_manager.cc:1347] Generated new cluster ID: bc3394d1b4a74a56b5b90e9295009288
I20260810 03:29:20.656044 17085 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 03:29:20.666500 17085 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 03:29:20.667760 17085 catalog_manager.cc:1504] Loading token signing keys...
I20260810 03:29:20.682415 17085 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 16bc7c23a5714a62a005b8dcf5969876: Generated new TSK 0
I20260810 03:29:20.683624 17085 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 03:29:20.694996 28292 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iBH5/build/asan/bin/kudu pid 16936
I20260810 03:29:20.696092 17046 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:29:20.785880 28292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iBH5/build/asan/bin/kudu with pid 16936
2026-08-10T03:29:20Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (2107 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 03:29:20 dist-test-slave-vpvm krb5kdc[17099](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 03:29:20 dist-test-slave-vpvm krb5kdc[17099](info): set up 2 sockets
Aug 10 03:29:20 dist-test-slave-vpvm krb5kdc[17099](info): commencing operation
krb5kdc: starting...
W20260810 03:29:22.003988 28292 mini_kdc.cc:121] Time spent starting KDC: real 1.176s user 0.000s sys 0.041s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/krb5kdc/test-user.keytab.
Aug 10 03:29:22 dist-test-slave-vpvm krb5kdc[17099](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332562, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T03:29:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T03:29:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.161.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.161.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.161.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.62.keytab.
Entry for principal kudu/127.27.161.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.62.keytab.
I20260810 03:29:22.288724 28292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iBH5/build/asan/bin/kudu
/tmp/dist-test-task24iBH5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.161.62:35141
--webserver_interface=127.27.161.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.62.keytab
--principal=kudu/127.27.161.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.161.20:36555
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.161.62:35141
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:29:22.624940 17119 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:29:22.625262 17119 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:29:22.625394 17119 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:29:22.632730 17119 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:29:22.632854 17119 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:29:22.632925 17119 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:29:22.633005 17119 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:29:22.633062 17119 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:29:22.646164 17119 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.161.20:36555
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.161.62:35141
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.62.keytab
--principal=kudu/127.27.161.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.161.62:35141
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.161.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:29:22.647832 17119 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:29:22.649449 17119 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:29:22.662886 17129 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:29:22.663070 17133 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:29:22.662866 17131 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:29:22.664523 17119 server_base.cc:1034] running on GCE node
I20260810 03:29:22.665356 17119 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:29:22.666889 17119 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:29:22.668115 17119 hybrid_clock.cc:648] HybridClock initialized: now 1786332562668052 us; error 50 us; skew 500 ppm
I20260810 03:29:22.668491 17119 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:29:22 dist-test-slave-vpvm krb5kdc[17099](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332562, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.161.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:29:22.671844 17119 init.cc:377] Logged in from keytab as kudu/127.27.161.62@KRBTEST.COM (short username kudu)
I20260810 03:29:22.673457 17119 webserver.cc:460] Webserver started at http://127.27.161.62:35053/ using document root <none> and password file <none>
I20260810 03:29:22.674283 17119 fs_manager.cc:359] Metadata directory not provided
I20260810 03:29:22.674435 17119 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:29:22.674775 17119 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:29:22.678058 17119 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/minicluster-data/master-0/data/instance:
uuid: "104edb053ce54ee1881fb2cbcace28cb"
format_stamp: "Formatted at 2026-08-10 03:29:22 on dist-test-slave-vpvm"
server_key: "aff4ca34f5f297d99297f8303e9c1a5b"
server_key_iv: "eab5ab9508a75442c91afc3bc3405470"
server_key_version: "encryptionkey@0"
I20260810 03:29:22.678886 17119 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/minicluster-data/master-0/wal/instance:
uuid: "104edb053ce54ee1881fb2cbcace28cb"
format_stamp: "Formatted at 2026-08-10 03:29:22 on dist-test-slave-vpvm"
server_key: "aff4ca34f5f297d99297f8303e9c1a5b"
server_key_iv: "eab5ab9508a75442c91afc3bc3405470"
server_key_version: "encryptionkey@0"
I20260810 03:29:22.684731 17119 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260810 03:29:22.688560 17141 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:29:22.689932 17119 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260810 03:29:22.690189 17119 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/minicluster-data/master-0/data,/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/minicluster-data/master-0/wal
uuid: "104edb053ce54ee1881fb2cbcace28cb"
format_stamp: "Formatted at 2026-08-10 03:29:22 on dist-test-slave-vpvm"
server_key: "aff4ca34f5f297d99297f8303e9c1a5b"
server_key_iv: "eab5ab9508a75442c91afc3bc3405470"
server_key_version: "encryptionkey@0"
I20260810 03:29:22.690400 17119 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:29:22.715606 17119 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:29:22.720301 17119 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:29:22.720746 17119 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:29:22.738121 17119 rpc_server.cc:307] RPC server started. Bound to: 127.27.161.62:35141
I20260810 03:29:22.738155 17228 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.161.62:35141 every 8 connection(s)
I20260810 03:29:22.739420 17119 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/minicluster-data/master-0/data/info.pb
I20260810 03:29:22.742970 17229 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:29:22.745887 28292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iBH5/build/asan/bin/kudu as pid 17119
I20260810 03:29:22.746133 28292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/minicluster-data/master-0/wal/instance
I20260810 03:29:22.746945 28292 external_mini_cluster.cc:1428] Setting key 85dee01edfd8bdf3b8bdd21a14b63071
I20260810 03:29:22.755203 17229 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 104edb053ce54ee1881fb2cbcace28cb: Bootstrap starting.
I20260810 03:29:22.758831 17229 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 104edb053ce54ee1881fb2cbcace28cb: Neither blocks nor log segments found. Creating new log.
I20260810 03:29:22.760071 17229 log.cc:824] T 00000000000000000000000000000000 P 104edb053ce54ee1881fb2cbcace28cb: Log is configured to *not* fsync() on all Append() calls
Aug 10 03:29:22 dist-test-slave-vpvm krb5kdc[17099](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332562, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.161.62@KRBTEST.COM
I20260810 03:29:22.763020 17229 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 104edb053ce54ee1881fb2cbcace28cb: No bootstrap required, opened a new log
I20260810 03:29:22.770465 17233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:29:22.749023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58230 (local address 127.27.161.62:35141)
0810 03:29:22.749723 (+ 700us) server_negotiation.cc:206] Beginning negotiation
0810 03:29:22.749733 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 03:29:22.749772 (+ 39us) server_negotiation.cc:407] Connection header received
0810 03:29:22.751410 (+ 1638us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:29:22.751431 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:29:22.751810 (+ 379us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:29:22.752120 (+ 310us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:29:22.753806 (+ 1686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:29:22.755112 (+ 1306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:29:22.756910 (+ 1798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:29:22.757551 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:29:22.757669 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:29:22.762290 (+ 4621us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:29:22.762301 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:29:22.762319 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:29:22.762366 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:29:22.766218 (+ 3852us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:29:22.766837 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:29:22.766849 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:29:22.766856 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:29:22.767063 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:29:22.767365 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:29:22.767370 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:29:22.767373 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:29:22.768033 (+ 660us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:29:22.768357 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0810 03:29:22.768665 (+ 308us) server_negotiation.cc:299] Negotiation successful
0810 03:29:22.769088 (+ 423us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":515,"thread_start_us":310,"threads_started":1}
I20260810 03:29:22.773165 17229 raft_consensus.cc:348] T 00000000000000000000000000000000 P 104edb053ce54ee1881fb2cbcace28cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "104edb053ce54ee1881fb2cbcace28cb" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 35141 } }
I20260810 03:29:22.773541 17229 raft_consensus.cc:374] T 00000000000000000000000000000000 P 104edb053ce54ee1881fb2cbcace28cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:29:22.773638 17229 raft_consensus.cc:729] T 00000000000000000000000000000000 P 104edb053ce54ee1881fb2cbcace28cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 104edb053ce54ee1881fb2cbcace28cb, State: Initialized, Role: FOLLOWER
I20260810 03:29:22.774201 17229 consensus_queue.cc:260] T 00000000000000000000000000000000 P 104edb053ce54ee1881fb2cbcace28cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "104edb053ce54ee1881fb2cbcace28cb" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 35141 } }
I20260810 03:29:22.774427 17229 raft_consensus.cc:388] T 00000000000000000000000000000000 P 104edb053ce54ee1881fb2cbcace28cb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 03:29:22.774549 17229 raft_consensus.cc:482] T 00000000000000000000000000000000 P 104edb053ce54ee1881fb2cbcace28cb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 03:29:22.774708 17229 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 104edb053ce54ee1881fb2cbcace28cb [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:29:22.776825 17229 raft_consensus.cc:504] T 00000000000000000000000000000000 P 104edb053ce54ee1881fb2cbcace28cb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "104edb053ce54ee1881fb2cbcace28cb" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 35141 } }
I20260810 03:29:22.777297 17229 leader_election.cc:302] T 00000000000000000000000000000000 P 104edb053ce54ee1881fb2cbcace28cb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 104edb053ce54ee1881fb2cbcace28cb; no voters:
I20260810 03:29:22.777853 17229 leader_election.cc:288] T 00000000000000000000000000000000 P 104edb053ce54ee1881fb2cbcace28cb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 03:29:22.778013 17235 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 104edb053ce54ee1881fb2cbcace28cb [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:29:22.778656 17235 raft_consensus.cc:686] T 00000000000000000000000000000000 P 104edb053ce54ee1881fb2cbcace28cb [term 1 LEADER]: Becoming Leader. State: Replica: 104edb053ce54ee1881fb2cbcace28cb, State: Running, Role: LEADER
I20260810 03:29:22.779251 17235 consensus_queue.cc:237] T 00000000000000000000000000000000 P 104edb053ce54ee1881fb2cbcace28cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "104edb053ce54ee1881fb2cbcace28cb" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 35141 } }
I20260810 03:29:22.780048 17229 sys_catalog.cc:564] T 00000000000000000000000000000000 P 104edb053ce54ee1881fb2cbcace28cb [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:29:22.782120 17236 sys_catalog.cc:455] T 00000000000000000000000000000000 P 104edb053ce54ee1881fb2cbcace28cb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "104edb053ce54ee1881fb2cbcace28cb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "104edb053ce54ee1881fb2cbcace28cb" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 35141 } } }
I20260810 03:29:22.782519 17236 sys_catalog.cc:458] T 00000000000000000000000000000000 P 104edb053ce54ee1881fb2cbcace28cb [sys.catalog]: This master's current role is: LEADER
I20260810 03:29:22.782433 17238 sys_catalog.cc:455] T 00000000000000000000000000000000 P 104edb053ce54ee1881fb2cbcace28cb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 104edb053ce54ee1881fb2cbcace28cb. Latest consensus state: current_term: 1 leader_uuid: "104edb053ce54ee1881fb2cbcace28cb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "104edb053ce54ee1881fb2cbcace28cb" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 35141 } } }
I20260810 03:29:22.782783 17238 sys_catalog.cc:458] T 00000000000000000000000000000000 P 104edb053ce54ee1881fb2cbcace28cb [sys.catalog]: This master's current role is: LEADER
I20260810 03:29:22.785070 17248 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 03:29:22.789443 17248 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 03:29:22.798190 17248 catalog_manager.cc:1347] Generated new cluster ID: 04b129a66593464187ede0ccb2dd4272
I20260810 03:29:22.798290 17248 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 03:29:22.814209 17248 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 03:29:22.815529 17248 catalog_manager.cc:1504] Loading token signing keys...
I20260810 03:29:22.823757 17248 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 104edb053ce54ee1881fb2cbcace28cb: Generated new TSK 0
I20260810 03:29:22.824573 17248 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.161.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.161.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.161.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.1.keytab.
Entry for principal kudu/127.27.161.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.1.keytab.
I20260810 03:29:22.899598 28292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iBH5/build/asan/bin/kudu
/tmp/dist-test-task24iBH5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.161.1:0
--local_ip_for_outbound_sockets=127.27.161.1
--webserver_interface=127.27.161.1
--webserver_port=0
--tserver_master_addrs=127.27.161.62:35141
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.1.keytab
--principal=kudu/127.27.161.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.161.20:36555
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:29:23.238570 17269 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:29:23.238826 17269 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:29:23.238960 17269 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:29:23.245991 17269 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:29:23.246101 17269 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:29:23.246253 17269 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.161.1
I20260810 03:29:23.255708 17269 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.161.20:36555
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.1.keytab
--principal=kudu/127.27.161.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.161.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.161.1
--webserver_port=0
--tserver_master_addrs=127.27.161.62:35141
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.161.1
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:29:23.256912 17269 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:29:23.258306 17269 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:29:23.270385 17275 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:29:23.270478 17280 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:29:23.271173 17276 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:29:23.272017 17269 server_base.cc:1034] running on GCE node
I20260810 03:29:23.272619 17269 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:29:23.273567 17269 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:29:23.274765 17269 hybrid_clock.cc:648] HybridClock initialized: now 1786332563274706 us; error 52 us; skew 500 ppm
I20260810 03:29:23.275084 17269 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:29:23 dist-test-slave-vpvm krb5kdc[17099](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332563, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.161.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:29:23.278079 17269 init.cc:377] Logged in from keytab as kudu/127.27.161.1@KRBTEST.COM (short username kudu)
I20260810 03:29:23.279467 17269 webserver.cc:460] Webserver started at http://127.27.161.1:37695/ using document root <none> and password file <none>
I20260810 03:29:23.280124 17269 fs_manager.cc:359] Metadata directory not provided
I20260810 03:29:23.280224 17269 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:29:23.280493 17269 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:29:23.283566 17269 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/minicluster-data/ts-0/data/instance:
uuid: "868802ad74164cc8b84e264576a9bb05"
format_stamp: "Formatted at 2026-08-10 03:29:23 on dist-test-slave-vpvm"
server_key: "844fdd946a70ccfd1a823157e16e41cb"
server_key_iv: "c4bddeddb2a89fbf0690af36a364c10e"
server_key_version: "encryptionkey@0"
I20260810 03:29:23.284320 17269 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/minicluster-data/ts-0/wal/instance:
uuid: "868802ad74164cc8b84e264576a9bb05"
format_stamp: "Formatted at 2026-08-10 03:29:23 on dist-test-slave-vpvm"
server_key: "844fdd946a70ccfd1a823157e16e41cb"
server_key_iv: "c4bddeddb2a89fbf0690af36a364c10e"
server_key_version: "encryptionkey@0"
I20260810 03:29:23.289665 17269 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260810 03:29:23.293052 17291 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:29:23.294445 17269 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 03:29:23.294595 17269 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/minicluster-data/ts-0/data,/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/minicluster-data/ts-0/wal
uuid: "868802ad74164cc8b84e264576a9bb05"
format_stamp: "Formatted at 2026-08-10 03:29:23 on dist-test-slave-vpvm"
server_key: "844fdd946a70ccfd1a823157e16e41cb"
server_key_iv: "c4bddeddb2a89fbf0690af36a364c10e"
server_key_version: "encryptionkey@0"
I20260810 03:29:23.294792 17269 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:29:23.308080 17269 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:29:23.312177 17269 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:29:23.312456 17269 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:29:23.313385 17269 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:29:23.314801 17269 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:29:23.314880 17269 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:29:23.315009 17269 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:29:23.315062 17269 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:29:23.345642 17269 rpc_server.cc:307] RPC server started. Bound to: 127.27.161.1:37585
I20260810 03:29:23.345678 17463 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.161.1:37585 every 8 connection(s)
I20260810 03:29:23.346925 17269 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/minicluster-data/ts-0/data/info.pb
I20260810 03:29:23.350829 28292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iBH5/build/asan/bin/kudu as pid 17269
I20260810 03:29:23.351064 28292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/minicluster-data/ts-0/wal/instance
I20260810 03:29:23.351927 28292 external_mini_cluster.cc:1428] Setting key ae65f7be405ae6d730a81b7dcb446be1
Aug 10 03:29:23 dist-test-slave-vpvm krb5kdc[17099](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332563, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.161.1@KRBTEST.COM for kudu/127.27.161.62@KRBTEST.COM
I20260810 03:29:23.368299 17468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:29:23.349391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.161.1:54663 (local address 127.27.161.62:35141)
0810 03:29:23.349869 (+ 478us) server_negotiation.cc:206] Beginning negotiation
0810 03:29:23.349876 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:29:23.351034 (+ 1158us) server_negotiation.cc:407] Connection header received
0810 03:29:23.353270 (+ 2236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:29:23.353276 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:29:23.353351 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:29:23.353469 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:29:23.355342 (+ 1873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:29:23.356052 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:29:23.358151 (+ 2099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:29:23.358540 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:29:23.358651 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:29:23.362739 (+ 4088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:29:23.362745 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:29:23.362748 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:29:23.362782 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:29:23.365219 (+ 2437us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:29:23.366042 (+ 823us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:29:23.366046 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:29:23.366049 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:29:23.366168 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:29:23.366583 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:29:23.366585 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:29:23.366587 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:29:23.366855 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:29:23.366973 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0810 03:29:23.367709 (+ 736us) server_negotiation.cc:299] Negotiation successful
0810 03:29:23.367938 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":349,"thread_start_us":231,"threads_started":1}
I20260810 03:29:23.369027 17469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:29:23.349981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:35141 (local address 127.27.161.1:54663)
0810 03:29:23.350808 (+ 827us) negotiation.cc:107] Waiting for socket to connect
0810 03:29:23.350860 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0810 03:29:23.353010 (+ 2150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:29:23.353666 (+ 656us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:29:23.353674 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:29:23.354352 (+ 678us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:29:23.355169 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:29:23.355195 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:29:23.356223 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:29:23.356228 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:29:23.358011 (+ 1783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:29:23.358027 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:29:23.358773 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:29:23.358780 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:29:23.359058 (+ 278us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:29:23.359756 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:29:23.359785 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:29:23.362486 (+ 2701us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:29:23.365361 (+ 2875us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:29:23.365371 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:29:23.365387 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:29:23.365909 (+ 522us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:29:23.366259 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:29:23.366263 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:29:23.366266 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:29:23.366495 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:29:23.366977 (+ 482us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:29:23.366985 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:29:23.367322 (+ 337us) client_negotiation.cc:769] Sending connection context
0810 03:29:23.367553 (+ 231us) client_negotiation.cc:240] Negotiation successful
0810 03:29:23.367946 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":534,"thread_start_us":295,"threads_started":1}
I20260810 03:29:23.370589 17464 heartbeater.cc:347] Connected to a master server at 127.27.161.62:35141
I20260810 03:29:23.370913 17464 heartbeater.cc:464] Registering TS with master...
I20260810 03:29:23.371848 17464 heartbeater.cc:511] Master 127.27.161.62:35141 requested a full tablet report, sending...
I20260810 03:29:23.374004 17174 ts_manager.cc:194] Registered new tserver with Master: 868802ad74164cc8b84e264576a9bb05 (127.27.161.1:37585)
I20260810 03:29:23.375435 17174 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.161.1@KRBTEST.COM'} at 127.27.161.1:54663
WARNING: no policy specified for kudu/127.27.161.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.161.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.161.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.2.keytab.
Entry for principal kudu/127.27.161.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.2.keytab.
I20260810 03:29:23.421017 28292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iBH5/build/asan/bin/kudu
/tmp/dist-test-task24iBH5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.161.2:0
--local_ip_for_outbound_sockets=127.27.161.2
--webserver_interface=127.27.161.2
--webserver_port=0
--tserver_master_addrs=127.27.161.62:35141
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.2.keytab
--principal=kudu/127.27.161.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.161.20:36555
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:29:23.754066 17473 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:29:23.754333 17473 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:29:23.754437 17473 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:29:23.761679 17473 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:29:23.761806 17473 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:29:23.761929 17473 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.161.2
I20260810 03:29:23.771776 17473 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.161.20:36555
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.2.keytab
--principal=kudu/127.27.161.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.161.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.161.2
--webserver_port=0
--tserver_master_addrs=127.27.161.62:35141
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.161.2
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:29:23.772993 17473 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:29:23.774307 17473 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:29:23.787258 17482 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:29:23.787223 17484 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:29:23.787314 17481 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:29:23.787963 17473 server_base.cc:1034] running on GCE node
I20260810 03:29:23.788606 17473 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:29:23.789565 17473 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:29:23.790767 17473 hybrid_clock.cc:648] HybridClock initialized: now 1786332563790716 us; error 38 us; skew 500 ppm
I20260810 03:29:23.791066 17473 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:29:23 dist-test-slave-vpvm krb5kdc[17099](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332563, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.161.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:29:23.794006 17473 init.cc:377] Logged in from keytab as kudu/127.27.161.2@KRBTEST.COM (short username kudu)
I20260810 03:29:23.795452 17473 webserver.cc:460] Webserver started at http://127.27.161.2:34879/ using document root <none> and password file <none>
I20260810 03:29:23.796124 17473 fs_manager.cc:359] Metadata directory not provided
I20260810 03:29:23.796265 17473 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:29:23.796602 17473 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:29:23.799757 17473 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/minicluster-data/ts-1/data/instance:
uuid: "6f9c1f7594aa43e5b1a4e3bd9dcdf7a1"
format_stamp: "Formatted at 2026-08-10 03:29:23 on dist-test-slave-vpvm"
server_key: "8ae896ece668e6bb6925e22796815542"
server_key_iv: "ff012c096deb7a32414389673d403323"
server_key_version: "encryptionkey@0"
I20260810 03:29:23.800541 17473 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/minicluster-data/ts-1/wal/instance:
uuid: "6f9c1f7594aa43e5b1a4e3bd9dcdf7a1"
format_stamp: "Formatted at 2026-08-10 03:29:23 on dist-test-slave-vpvm"
server_key: "8ae896ece668e6bb6925e22796815542"
server_key_iv: "ff012c096deb7a32414389673d403323"
server_key_version: "encryptionkey@0"
I20260810 03:29:23.805666 17473 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260810 03:29:23.809054 17493 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:29:23.810365 17473 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 03:29:23.810525 17473 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/minicluster-data/ts-1/data,/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/minicluster-data/ts-1/wal
uuid: "6f9c1f7594aa43e5b1a4e3bd9dcdf7a1"
format_stamp: "Formatted at 2026-08-10 03:29:23 on dist-test-slave-vpvm"
server_key: "8ae896ece668e6bb6925e22796815542"
server_key_iv: "ff012c096deb7a32414389673d403323"
server_key_version: "encryptionkey@0"
I20260810 03:29:23.810708 17473 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:29:23.855075 17473 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:29:23.859050 17473 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:29:23.859669 17473 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:29:23.860669 17473 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:29:23.862032 17473 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:29:23.862120 17473 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:29:23.862241 17473 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:29:23.862296 17473 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:29:23.893302 17473 rpc_server.cc:307] RPC server started. Bound to: 127.27.161.2:45581
I20260810 03:29:23.893349 17674 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.161.2:45581 every 8 connection(s)
I20260810 03:29:23.894560 17473 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/minicluster-data/ts-1/data/info.pb
I20260810 03:29:23.898510 28292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iBH5/build/asan/bin/kudu as pid 17473
I20260810 03:29:23.898722 28292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/minicluster-data/ts-1/wal/instance
I20260810 03:29:23.899236 28292 external_mini_cluster.cc:1428] Setting key a0c2bcc6cc42cc91430fc80dbcab7f68
Aug 10 03:29:23 dist-test-slave-vpvm krb5kdc[17099](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332563, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.161.2@KRBTEST.COM for kudu/127.27.161.62@KRBTEST.COM
I20260810 03:29:23.914794 17678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:29:23.897025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.161.2:52103 (local address 127.27.161.62:35141)
0810 03:29:23.897456 (+ 431us) server_negotiation.cc:206] Beginning negotiation
0810 03:29:23.897460 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:29:23.898382 (+ 922us) server_negotiation.cc:407] Connection header received
0810 03:29:23.900436 (+ 2054us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:29:23.900440 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:29:23.900505 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:29:23.900594 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:29:23.902366 (+ 1772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:29:23.903388 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:29:23.904904 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:29:23.905446 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:29:23.905561 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:29:23.909040 (+ 3479us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:29:23.909045 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:29:23.909048 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:29:23.909081 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:29:23.911555 (+ 2474us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:29:23.912361 (+ 806us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:29:23.912364 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:29:23.912367 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:29:23.912492 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:29:23.912927 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:29:23.912930 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:29:23.912932 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:29:23.913182 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:29:23.913287 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0810 03:29:23.914157 (+ 870us) server_negotiation.cc:299] Negotiation successful
0810 03:29:23.914379 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":331,"thread_start_us":224,"threads_started":1}
I20260810 03:29:23.915660 17680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:29:23.897464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:35141 (local address 127.27.161.2:52103)
0810 03:29:23.898229 (+ 765us) negotiation.cc:107] Waiting for socket to connect
0810 03:29:23.898268 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0810 03:29:23.900161 (+ 1893us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:29:23.900864 (+ 703us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:29:23.900876 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:29:23.901564 (+ 688us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:29:23.902199 (+ 635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:29:23.902220 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:29:23.903519 (+ 1299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:29:23.903522 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:29:23.904762 (+ 1240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:29:23.904776 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:29:23.905642 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:29:23.905647 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:29:23.905837 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:29:23.906484 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:29:23.906508 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:29:23.908789 (+ 2281us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:29:23.911693 (+ 2904us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:29:23.911702 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:29:23.911717 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:29:23.912257 (+ 540us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:29:23.912591 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:29:23.912595 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:29:23.912598 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:29:23.912833 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:29:23.913292 (+ 459us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:29:23.913301 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:29:23.913738 (+ 437us) client_negotiation.cc:769] Sending connection context
0810 03:29:23.914019 (+ 281us) client_negotiation.cc:240] Negotiation successful
0810 03:29:23.914473 (+ 454us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":512,"thread_start_us":261,"threads_started":1}
I20260810 03:29:23.917083 17676 heartbeater.cc:347] Connected to a master server at 127.27.161.62:35141
I20260810 03:29:23.917415 17676 heartbeater.cc:464] Registering TS with master...
I20260810 03:29:23.918288 17676 heartbeater.cc:511] Master 127.27.161.62:35141 requested a full tablet report, sending...
I20260810 03:29:23.919899 17174 ts_manager.cc:194] Registered new tserver with Master: 6f9c1f7594aa43e5b1a4e3bd9dcdf7a1 (127.27.161.2:45581)
I20260810 03:29:23.920704 17174 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.161.2@KRBTEST.COM'} at 127.27.161.2:52103
WARNING: no policy specified for kudu/127.27.161.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.161.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.161.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.3.keytab.
Entry for principal kudu/127.27.161.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.3.keytab.
I20260810 03:29:23.962996 28292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iBH5/build/asan/bin/kudu
/tmp/dist-test-task24iBH5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.161.3:0
--local_ip_for_outbound_sockets=127.27.161.3
--webserver_interface=127.27.161.3
--webserver_port=0
--tserver_master_addrs=127.27.161.62:35141
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.3.keytab
--principal=kudu/127.27.161.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.161.20:36555
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:29:24.296516 17684 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:29:24.296797 17684 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:29:24.296929 17684 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:29:24.304039 17684 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:29:24.304141 17684 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:29:24.304291 17684 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.161.3
I20260810 03:29:24.313887 17684 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.161.20:36555
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.3.keytab
--principal=kudu/127.27.161.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.161.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.161.3
--webserver_port=0
--tserver_master_addrs=127.27.161.62:35141
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.161.3
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:29:24.315292 17684 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:29:24.316582 17684 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:29:24.329309 17694 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:29:24.329557 17692 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:29:24.329350 17696 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:29:24.330235 17684 server_base.cc:1034] running on GCE node
I20260810 03:29:24.330806 17684 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:29:24.331687 17684 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:29:24.332870 17684 hybrid_clock.cc:648] HybridClock initialized: now 1786332564332814 us; error 48 us; skew 500 ppm
I20260810 03:29:24.333158 17684 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:29:24 dist-test-slave-vpvm krb5kdc[17099](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332564, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.161.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:29:24.336190 17684 init.cc:377] Logged in from keytab as kudu/127.27.161.3@KRBTEST.COM (short username kudu)
I20260810 03:29:24.337590 17684 webserver.cc:460] Webserver started at http://127.27.161.3:33479/ using document root <none> and password file <none>
I20260810 03:29:24.338299 17684 fs_manager.cc:359] Metadata directory not provided
I20260810 03:29:24.338410 17684 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:29:24.338706 17684 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:29:24.341591 17684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/minicluster-data/ts-2/data/instance:
uuid: "386dbc17f7974e99a6a7500ccc1d16f8"
format_stamp: "Formatted at 2026-08-10 03:29:24 on dist-test-slave-vpvm"
server_key: "c10c20a349ff82746b8dc4eca2a2671b"
server_key_iv: "f0e46891c61ae3b300c0791780f9daeb"
server_key_version: "encryptionkey@0"
I20260810 03:29:24.342387 17684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/minicluster-data/ts-2/wal/instance:
uuid: "386dbc17f7974e99a6a7500ccc1d16f8"
format_stamp: "Formatted at 2026-08-10 03:29:24 on dist-test-slave-vpvm"
server_key: "c10c20a349ff82746b8dc4eca2a2671b"
server_key_iv: "f0e46891c61ae3b300c0791780f9daeb"
server_key_version: "encryptionkey@0"
I20260810 03:29:24.347626 17684 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260810 03:29:24.351043 17705 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:29:24.352262 17684 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260810 03:29:24.352407 17684 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/minicluster-data/ts-2/data,/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/minicluster-data/ts-2/wal
uuid: "386dbc17f7974e99a6a7500ccc1d16f8"
format_stamp: "Formatted at 2026-08-10 03:29:24 on dist-test-slave-vpvm"
server_key: "c10c20a349ff82746b8dc4eca2a2671b"
server_key_iv: "f0e46891c61ae3b300c0791780f9daeb"
server_key_version: "encryptionkey@0"
I20260810 03:29:24.352598 17684 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:29:24.378826 17464 heartbeater.cc:503] Master 127.27.161.62:35141 was elected leader, sending a full tablet report...
I20260810 03:29:24.386070 17684 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:29:24.390383 17684 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:29:24.390650 17684 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:29:24.391575 17684 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:29:24.392902 17684 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:29:24.392977 17684 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:29:24.393076 17684 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:29:24.393121 17684 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:29:24.424440 17684 rpc_server.cc:307] RPC server started. Bound to: 127.27.161.3:46513
I20260810 03:29:24.424486 17879 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.161.3:46513 every 8 connection(s)
I20260810 03:29:24.425709 17684 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/minicluster-data/ts-2/data/info.pb
I20260810 03:29:24.429685 28292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iBH5/build/asan/bin/kudu as pid 17684
I20260810 03:29:24.429872 28292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/minicluster-data/ts-2/wal/instance
I20260810 03:29:24.430330 28292 external_mini_cluster.cc:1428] Setting key eb260a8963d5a85e41a7eec688884d31
Aug 10 03:29:24 dist-test-slave-vpvm krb5kdc[17099](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332564, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.161.3@KRBTEST.COM for kudu/127.27.161.62@KRBTEST.COM
I20260810 03:29:24.444877 17883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:29:24.428286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.161.3:56935 (local address 127.27.161.62:35141)
0810 03:29:24.428712 (+ 426us) server_negotiation.cc:206] Beginning negotiation
0810 03:29:24.428717 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:29:24.429670 (+ 953us) server_negotiation.cc:407] Connection header received
0810 03:29:24.431708 (+ 2038us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:29:24.431714 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:29:24.431799 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:29:24.431892 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:29:24.433321 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:29:24.434148 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:29:24.435647 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:29:24.436046 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:29:24.436166 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:29:24.439483 (+ 3317us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:29:24.439488 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:29:24.439491 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:29:24.439526 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:29:24.441833 (+ 2307us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:29:24.442538 (+ 705us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:29:24.442543 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:29:24.442546 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:29:24.442664 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:29:24.443073 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:29:24.443076 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:29:24.443078 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:29:24.443371 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:29:24.443514 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0810 03:29:24.444243 (+ 729us) server_negotiation.cc:299] Negotiation successful
0810 03:29:24.444469 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317,"thread_start_us":194,"threads_started":1}
I20260810 03:29:24.445775 17884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:29:24.428849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:35141 (local address 127.27.161.3:56935)
0810 03:29:24.429522 (+ 673us) negotiation.cc:107] Waiting for socket to connect
0810 03:29:24.429558 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0810 03:29:24.431420 (+ 1862us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:29:24.432132 (+ 712us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:29:24.432140 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:29:24.432618 (+ 478us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:29:24.433159 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:29:24.433176 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:29:24.434305 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:29:24.434309 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:29:24.435507 (+ 1198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:29:24.435519 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:29:24.436270 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:29:24.436278 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:29:24.436493 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:29:24.437125 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:29:24.437154 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:29:24.439269 (+ 2115us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:29:24.441989 (+ 2720us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:29:24.441998 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:29:24.442015 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:29:24.442420 (+ 405us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:29:24.442767 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:29:24.442773 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:29:24.442776 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:29:24.442956 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:29:24.443497 (+ 541us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:29:24.443505 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:29:24.443851 (+ 346us) client_negotiation.cc:769] Sending connection context
0810 03:29:24.444092 (+ 241us) client_negotiation.cc:240] Negotiation successful
0810 03:29:24.444638 (+ 546us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":267,"threads_started":1}
I20260810 03:29:24.447033 17880 heartbeater.cc:347] Connected to a master server at 127.27.161.62:35141
I20260810 03:29:24.447307 17880 heartbeater.cc:464] Registering TS with master...
I20260810 03:29:24.447990 17880 heartbeater.cc:511] Master 127.27.161.62:35141 requested a full tablet report, sending...
I20260810 03:29:24.449298 17174 ts_manager.cc:194] Registered new tserver with Master: 386dbc17f7974e99a6a7500ccc1d16f8 (127.27.161.3:46513)
I20260810 03:29:24.450121 17174 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.161.3@KRBTEST.COM'} at 127.27.161.3:56935
I20260810 03:29:24.457482 28292 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 03:29:24.473397 17883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:29:24.461884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58238 (local address 127.27.161.62:35141)
0810 03:29:24.462057 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0810 03:29:24.462061 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:29:24.462498 (+ 437us) server_negotiation.cc:407] Connection header received
0810 03:29:24.462625 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:29:24.462628 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:29:24.462684 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:29:24.462758 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:29:24.464010 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:29:24.464719 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:29:24.465975 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:29:24.466376 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:29:24.466466 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:29:24.468932 (+ 2466us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:29:24.468937 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:29:24.468940 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:29:24.468971 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:29:24.471315 (+ 2344us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:29:24.471915 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:29:24.471918 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:29:24.471921 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:29:24.472023 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:29:24.472307 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:29:24.472310 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:29:24.472311 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:29:24.472605 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:29:24.472780 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0810 03:29:24.473016 (+ 236us) server_negotiation.cc:299] Negotiation successful
0810 03:29:24.473183 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260810 03:29:24.477205 17174 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58238:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 03:29:24.484011 17174 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 03:29:24.531718 17907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:29:24.495982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.2:45581 (local address 127.0.0.1:39166)
0810 03:29:24.523285 (+ 27303us) negotiation.cc:107] Waiting for socket to connect
0810 03:29:24.523305 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 03:29:24.523425 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:29:24.524413 (+ 988us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:29:24.524419 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:29:24.524454 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:29:24.524923 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:29:24.524937 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:29:24.526731 (+ 1794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:29:24.526736 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:29:24.529218 (+ 2482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:29:24.529236 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:29:24.530707 (+ 1471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:29:24.530714 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:29:24.530891 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:29:24.530941 (+ 50us) client_negotiation.cc:769] Sending connection context
0810 03:29:24.531063 (+ 122us) client_negotiation.cc:240] Negotiation successful
0810 03:29:24.531267 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27170,"mutex_wait_us":47,"thread_start_us":27007,"threads_started":1}
I20260810 03:29:24.531767 17910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:29:24.496523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39166 (local address 127.27.161.2:45581)
0810 03:29:24.522104 (+ 25581us) server_negotiation.cc:206] Beginning negotiation
0810 03:29:24.522113 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 03:29:24.523548 (+ 1435us) server_negotiation.cc:407] Connection header received
0810 03:29:24.523657 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:29:24.523664 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:29:24.523922 (+ 258us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:29:24.524116 (+ 194us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:29:24.525156 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:29:24.526581 (+ 1425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:29:24.529388 (+ 2807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:29:24.530495 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:29:24.530623 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:29:24.531229 (+ 606us) server_negotiation.cc:1035] Waiting for connection context
0810 03:29:24.531303 (+ 74us) server_negotiation.cc:299] Negotiation successful
0810 03:29:24.531472 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25363,"thread_start_us":25249,"threads_started":1}
I20260810 03:29:24.532557 17909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:29:24.495739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.1:37585 (local address 127.0.0.1:36946)
0810 03:29:24.521091 (+ 25352us) negotiation.cc:107] Waiting for socket to connect
0810 03:29:24.521153 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0810 03:29:24.521411 (+ 258us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:29:24.525677 (+ 4266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:29:24.525680 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:29:24.525699 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:29:24.526019 (+ 320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:29:24.526028 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:29:24.527821 (+ 1793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:29:24.527825 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:29:24.529591 (+ 1766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:29:24.529606 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:29:24.531460 (+ 1854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:29:24.531467 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:29:24.531652 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:29:24.531688 (+ 36us) client_negotiation.cc:769] Sending connection context
0810 03:29:24.531758 (+ 70us) client_negotiation.cc:240] Negotiation successful
0810 03:29:24.532142 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24993,"mutex_wait_us":4,"thread_start_us":29433,"threads_started":1}
I20260810 03:29:24.533018 17905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:29:24.495778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36946 (local address 127.27.161.1:37585)
0810 03:29:24.524448 (+ 28670us) server_negotiation.cc:206] Beginning negotiation
0810 03:29:24.524457 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 03:29:24.524492 (+ 35us) server_negotiation.cc:407] Connection header received
0810 03:29:24.524612 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:29:24.524620 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:29:24.524833 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:29:24.526192 (+ 1359us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:29:24.526454 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:29:24.527671 (+ 1217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:29:24.529758 (+ 2087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:29:24.531212 (+ 1454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:29:24.532158 (+ 946us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:29:24.532394 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0810 03:29:24.532461 (+ 67us) server_negotiation.cc:299] Negotiation successful
0810 03:29:24.532714 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28475,"thread_start_us":29400,"threads_started":1}
I20260810 03:29:24.533255 17906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:29:24.496102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.3:46513 (local address 127.0.0.1:36376)
0810 03:29:24.525418 (+ 29316us) negotiation.cc:107] Waiting for socket to connect
0810 03:29:24.525434 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 03:29:24.525547 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:29:24.526156 (+ 609us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:29:24.526160 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:29:24.526184 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:29:24.526520 (+ 336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:29:24.526533 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:29:24.528475 (+ 1942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:29:24.528481 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:29:24.530492 (+ 2011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:29:24.530517 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:29:24.532561 (+ 2044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:29:24.532568 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:29:24.532720 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:29:24.532751 (+ 31us) client_negotiation.cc:769] Sending connection context
0810 03:29:24.532839 (+ 88us) client_negotiation.cc:240] Negotiation successful
0810 03:29:24.532958 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29213,"thread_start_us":26049,"threads_started":1}
I20260810 03:29:24.533694 17911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:29:24.496664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36376 (local address 127.27.161.3:46513)
0810 03:29:24.520137 (+ 23473us) server_negotiation.cc:206] Beginning negotiation
0810 03:29:24.520148 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 03:29:24.525539 (+ 5391us) server_negotiation.cc:407] Connection header received
0810 03:29:24.525741 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:29:24.525749 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:29:24.525934 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:29:24.526155 (+ 221us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:29:24.526699 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:29:24.528275 (+ 1576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:29:24.530692 (+ 2417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:29:24.532051 (+ 1359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:29:24.532168 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:29:24.532393 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0810 03:29:24.533250 (+ 857us) server_negotiation.cc:299] Negotiation successful
0810 03:29:24.533409 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23202,"thread_start_us":23503,"threads_started":1}
I20260810 03:29:24.536608 17367 tablet_service.cc:1467] Processing CreateTablet for tablet 8cc62b872e2c4a6790d75ff4ecd560b0 (DEFAULT_TABLE table=test-table [id=575497a7045e47538c60ac043e0364c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:29:24.536670 17578 tablet_service.cc:1467] Processing CreateTablet for tablet 8cc62b872e2c4a6790d75ff4ecd560b0 (DEFAULT_TABLE table=test-table [id=575497a7045e47538c60ac043e0364c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:29:24.536882 17778 tablet_service.cc:1467] Processing CreateTablet for tablet 8cc62b872e2c4a6790d75ff4ecd560b0 (DEFAULT_TABLE table=test-table [id=575497a7045e47538c60ac043e0364c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:29:24.538035 17367 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8cc62b872e2c4a6790d75ff4ecd560b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:29:24.538084 17578 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8cc62b872e2c4a6790d75ff4ecd560b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:29:24.538086 17778 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8cc62b872e2c4a6790d75ff4ecd560b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:29:24.549118 17913 tablet_bootstrap.cc:492] T 8cc62b872e2c4a6790d75ff4ecd560b0 P 868802ad74164cc8b84e264576a9bb05: Bootstrap starting.
I20260810 03:29:24.552138 17913 tablet_bootstrap.cc:654] T 8cc62b872e2c4a6790d75ff4ecd560b0 P 868802ad74164cc8b84e264576a9bb05: Neither blocks nor log segments found. Creating new log.
I20260810 03:29:24.552969 17915 tablet_bootstrap.cc:492] T 8cc62b872e2c4a6790d75ff4ecd560b0 P 6f9c1f7594aa43e5b1a4e3bd9dcdf7a1: Bootstrap starting.
I20260810 03:29:24.553380 17913 log.cc:824] T 8cc62b872e2c4a6790d75ff4ecd560b0 P 868802ad74164cc8b84e264576a9bb05: Log is configured to *not* fsync() on all Append() calls
I20260810 03:29:24.555946 17914 tablet_bootstrap.cc:492] T 8cc62b872e2c4a6790d75ff4ecd560b0 P 386dbc17f7974e99a6a7500ccc1d16f8: Bootstrap starting.
I20260810 03:29:24.556660 17915 tablet_bootstrap.cc:654] T 8cc62b872e2c4a6790d75ff4ecd560b0 P 6f9c1f7594aa43e5b1a4e3bd9dcdf7a1: Neither blocks nor log segments found. Creating new log.
I20260810 03:29:24.557886 17915 log.cc:824] T 8cc62b872e2c4a6790d75ff4ecd560b0 P 6f9c1f7594aa43e5b1a4e3bd9dcdf7a1: Log is configured to *not* fsync() on all Append() calls
I20260810 03:29:24.559242 17914 tablet_bootstrap.cc:654] T 8cc62b872e2c4a6790d75ff4ecd560b0 P 386dbc17f7974e99a6a7500ccc1d16f8: Neither blocks nor log segments found. Creating new log.
I20260810 03:29:24.559733 17913 tablet_bootstrap.cc:492] T 8cc62b872e2c4a6790d75ff4ecd560b0 P 868802ad74164cc8b84e264576a9bb05: No bootstrap required, opened a new log
I20260810 03:29:24.559990 17913 ts_tablet_manager.cc:1397] T 8cc62b872e2c4a6790d75ff4ecd560b0 P 868802ad74164cc8b84e264576a9bb05: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.000s
I20260810 03:29:24.560448 17914 log.cc:824] T 8cc62b872e2c4a6790d75ff4ecd560b0 P 386dbc17f7974e99a6a7500ccc1d16f8: Log is configured to *not* fsync() on all Append() calls
I20260810 03:29:24.560976 17915 tablet_bootstrap.cc:492] T 8cc62b872e2c4a6790d75ff4ecd560b0 P 6f9c1f7594aa43e5b1a4e3bd9dcdf7a1: No bootstrap required, opened a new log
I20260810 03:29:24.561228 17915 ts_tablet_manager.cc:1397] T 8cc62b872e2c4a6790d75ff4ecd560b0 P 6f9c1f7594aa43e5b1a4e3bd9dcdf7a1: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260810 03:29:24.563380 17914 tablet_bootstrap.cc:492] T 8cc62b872e2c4a6790d75ff4ecd560b0 P 386dbc17f7974e99a6a7500ccc1d16f8: No bootstrap required, opened a new log
I20260810 03:29:24.563673 17914 ts_tablet_manager.cc:1397] T 8cc62b872e2c4a6790d75ff4ecd560b0 P 386dbc17f7974e99a6a7500ccc1d16f8: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260810 03:29:24.567484 17913 raft_consensus.cc:348] T 8cc62b872e2c4a6790d75ff4ecd560b0 P 868802ad74164cc8b84e264576a9bb05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "868802ad74164cc8b84e264576a9bb05" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 37585 } } peers { permanent_uuid: "6f9c1f7594aa43e5b1a4e3bd9dcdf7a1" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 45581 } } peers { permanent_uuid: "386dbc17f7974e99a6a7500ccc1d16f8" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 46513 } }
I20260810 03:29:24.567818 17913 raft_consensus.cc:374] T 8cc62b872e2c4a6790d75ff4ecd560b0 P 868802ad74164cc8b84e264576a9bb05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:29:24.567906 17913 raft_consensus.cc:729] T 8cc62b872e2c4a6790d75ff4ecd560b0 P 868802ad74164cc8b84e264576a9bb05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 868802ad74164cc8b84e264576a9bb05, State: Initialized, Role: FOLLOWER
I20260810 03:29:24.568437 17913 consensus_queue.cc:260] T 8cc62b872e2c4a6790d75ff4ecd560b0 P 868802ad74164cc8b84e264576a9bb05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "868802ad74164cc8b84e264576a9bb05" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 37585 } } peers { permanent_uuid: "6f9c1f7594aa43e5b1a4e3bd9dcdf7a1" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 45581 } } peers { permanent_uuid: "386dbc17f7974e99a6a7500ccc1d16f8" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 46513 } }
I20260810 03:29:24.570194 17913 ts_tablet_manager.cc:1428] T 8cc62b872e2c4a6790d75ff4ecd560b0 P 868802ad74164cc8b84e264576a9bb05: Time spent starting tablet: real 0.010s user 0.006s sys 0.006s
I20260810 03:29:24.571066 17915 raft_consensus.cc:348] T 8cc62b872e2c4a6790d75ff4ecd560b0 P 6f9c1f7594aa43e5b1a4e3bd9dcdf7a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "868802ad74164cc8b84e264576a9bb05" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 37585 } } peers { permanent_uuid: "6f9c1f7594aa43e5b1a4e3bd9dcdf7a1" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 45581 } } peers { permanent_uuid: "386dbc17f7974e99a6a7500ccc1d16f8" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 46513 } }
I20260810 03:29:24.571478 17915 raft_consensus.cc:374] T 8cc62b872e2c4a6790d75ff4ecd560b0 P 6f9c1f7594aa43e5b1a4e3bd9dcdf7a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:29:24.571599 17915 raft_consensus.cc:729] T 8cc62b872e2c4a6790d75ff4ecd560b0 P 6f9c1f7594aa43e5b1a4e3bd9dcdf7a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f9c1f7594aa43e5b1a4e3bd9dcdf7a1, State: Initialized, Role: FOLLOWER
I20260810 03:29:24.572189 17915 consensus_queue.cc:260] T 8cc62b872e2c4a6790d75ff4ecd560b0 P 6f9c1f7594aa43e5b1a4e3bd9dcdf7a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "868802ad74164cc8b84e264576a9bb05" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 37585 } } peers { permanent_uuid: "6f9c1f7594aa43e5b1a4e3bd9dcdf7a1" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 45581 } } peers { permanent_uuid: "386dbc17f7974e99a6a7500ccc1d16f8" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 46513 } }
I20260810 03:29:24.573570 17914 raft_consensus.cc:348] T 8cc62b872e2c4a6790d75ff4ecd560b0 P 386dbc17f7974e99a6a7500ccc1d16f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "868802ad74164cc8b84e264576a9bb05" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 37585 } } peers { permanent_uuid: "6f9c1f7594aa43e5b1a4e3bd9dcdf7a1" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 45581 } } peers { permanent_uuid: "386dbc17f7974e99a6a7500ccc1d16f8" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 46513 } }
I20260810 03:29:24.573932 17915 ts_tablet_manager.cc:1428] T 8cc62b872e2c4a6790d75ff4ecd560b0 P 6f9c1f7594aa43e5b1a4e3bd9dcdf7a1: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
I20260810 03:29:24.574019 17914 raft_consensus.cc:374] T 8cc62b872e2c4a6790d75ff4ecd560b0 P 386dbc17f7974e99a6a7500ccc1d16f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:29:24.574115 17914 raft_consensus.cc:729] T 8cc62b872e2c4a6790d75ff4ecd560b0 P 386dbc17f7974e99a6a7500ccc1d16f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 386dbc17f7974e99a6a7500ccc1d16f8, State: Initialized, Role: FOLLOWER
I20260810 03:29:24.574636 17676 heartbeater.cc:503] Master 127.27.161.62:35141 was elected leader, sending a full tablet report...
I20260810 03:29:24.574726 17914 consensus_queue.cc:260] T 8cc62b872e2c4a6790d75ff4ecd560b0 P 386dbc17f7974e99a6a7500ccc1d16f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "868802ad74164cc8b84e264576a9bb05" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 37585 } } peers { permanent_uuid: "6f9c1f7594aa43e5b1a4e3bd9dcdf7a1" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 45581 } } peers { permanent_uuid: "386dbc17f7974e99a6a7500ccc1d16f8" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 46513 } }
I20260810 03:29:24.575800 17880 heartbeater.cc:503] Master 127.27.161.62:35141 was elected leader, sending a full tablet report...
I20260810 03:29:24.577560 17914 ts_tablet_manager.cc:1428] T 8cc62b872e2c4a6790d75ff4ecd560b0 P 386dbc17f7974e99a6a7500ccc1d16f8: Time spent starting tablet: real 0.014s user 0.008s sys 0.004s
W20260810 03:29:24.599220 17467 tablet.cc:2369] T 8cc62b872e2c4a6790d75ff4ecd560b0 P 868802ad74164cc8b84e264576a9bb05: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 03:29:24.609984 17925 raft_consensus.cc:482] T 8cc62b872e2c4a6790d75ff4ecd560b0 P 6f9c1f7594aa43e5b1a4e3bd9dcdf7a1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 03:29:24.610312 17925 raft_consensus.cc:504] T 8cc62b872e2c4a6790d75ff4ecd560b0 P 6f9c1f7594aa43e5b1a4e3bd9dcdf7a1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "868802ad74164cc8b84e264576a9bb05" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 37585 } } peers { permanent_uuid: "6f9c1f7594aa43e5b1a4e3bd9dcdf7a1" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 45581 } } peers { permanent_uuid: "386dbc17f7974e99a6a7500ccc1d16f8" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 46513 } }
I20260810 03:29:24.612001 17925 leader_election.cc:288] T 8cc62b872e2c4a6790d75ff4ecd560b0 P 6f9c1f7594aa43e5b1a4e3bd9dcdf7a1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 868802ad74164cc8b84e264576a9bb05 (127.27.161.1:37585), 386dbc17f7974e99a6a7500ccc1d16f8 (127.27.161.3:46513)
I20260810 03:29:24.618799 17911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:29:24.612522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.161.2:53073 (local address 127.27.161.3:46513)
0810 03:29:24.612758 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0810 03:29:24.612765 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:29:24.613376 (+ 611us) server_negotiation.cc:407] Connection header received
0810 03:29:24.613534 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:29:24.613537 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:29:24.613619 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:29:24.613696 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:29:24.614410 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:29:24.615149 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:29:24.617078 (+ 1929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:29:24.617828 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:29:24.618018 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:29:24.618103 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0810 03:29:24.618437 (+ 334us) server_negotiation.cc:299] Negotiation successful
0810 03:29:24.618559 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260810 03:29:24.618805 17905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:29:24.612350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.161.2:44309 (local address 127.27.161.1:37585)
0810 03:29:24.612603 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0810 03:29:24.612611 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 03:29:24.613310 (+ 699us) server_negotiation.cc:407] Connection header received
0810 03:29:24.613509 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:29:24.613513 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:29:24.613577 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:29:24.613663 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:29:24.614429 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:29:24.615186 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:29:24.617085 (+ 1899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:29:24.617856 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:29:24.618030 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:29:24.618104 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0810 03:29:24.618434 (+ 330us) server_negotiation.cc:299] Negotiation successful
0810 03:29:24.618568 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106}
I20260810 03:29:24.619102 17929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:29:24.612222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.1:37585 (local address 127.27.161.2:44309)
0810 03:29:24.613195 (+ 973us) negotiation.cc:107] Waiting for socket to connect
0810 03:29:24.613211 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 03:29:24.613365 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:29:24.613683 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:29:24.613687 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:29:24.613713 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:29:24.614296 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:29:24.614309 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:29:24.615299 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:29:24.615305 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:29:24.616930 (+ 1625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:29:24.616943 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:29:24.618131 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:29:24.618138 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:29:24.618283 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:29:24.618314 (+ 31us) client_negotiation.cc:769] Sending connection context
0810 03:29:24.618393 (+ 79us) client_negotiation.cc:240] Negotiation successful
0810 03:29:24.618551 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":816,"mutex_wait_us":73,"thread_start_us":590,"threads_started":1}
I20260810 03:29:24.619268 17930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:29:24.612485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.3:46513 (local address 127.27.161.2:53073)
0810 03:29:24.613262 (+ 777us) negotiation.cc:107] Waiting for socket to connect
0810 03:29:24.613276 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 03:29:24.613404 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:29:24.613726 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:29:24.613731 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:29:24.613754 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:29:24.614252 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:29:24.614267 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:29:24.615294 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:29:24.615300 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:29:24.616931 (+ 1631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:29:24.616944 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:29:24.618101 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:29:24.618112 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:29:24.618272 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:29:24.618307 (+ 35us) client_negotiation.cc:769] Sending connection context
0810 03:29:24.618389 (+ 82us) client_negotiation.cc:240] Negotiation successful
0810 03:29:24.618548 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":672,"mutex_wait_us":84,"thread_start_us":446,"threads_started":1}
I20260810 03:29:24.620038 17401 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cc62b872e2c4a6790d75ff4ecd560b0" candidate_uuid: "6f9c1f7594aa43e5b1a4e3bd9dcdf7a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "868802ad74164cc8b84e264576a9bb05" is_pre_election: true
I20260810 03:29:24.620085 17808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cc62b872e2c4a6790d75ff4ecd560b0" candidate_uuid: "6f9c1f7594aa43e5b1a4e3bd9dcdf7a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "386dbc17f7974e99a6a7500ccc1d16f8" is_pre_election: true
I20260810 03:29:24.620512 17808 raft_consensus.cc:2454] T 8cc62b872e2c4a6790d75ff4ecd560b0 P 386dbc17f7974e99a6a7500ccc1d16f8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6f9c1f7594aa43e5b1a4e3bd9dcdf7a1 in term 0.
I20260810 03:29:24.620520 17401 raft_consensus.cc:2454] T 8cc62b872e2c4a6790d75ff4ecd560b0 P 868802ad74164cc8b84e264576a9bb05 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6f9c1f7594aa43e5b1a4e3bd9dcdf7a1 in term 0.
I20260810 03:29:24.621152 17499 leader_election.cc:302] T 8cc62b872e2c4a6790d75ff4ecd560b0 P 6f9c1f7594aa43e5b1a4e3bd9dcdf7a1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 386dbc17f7974e99a6a7500ccc1d16f8, 6f9c1f7594aa43e5b1a4e3bd9dcdf7a1, 868802ad74164cc8b84e264576a9bb05; no voters:
I20260810 03:29:24.621590 17925 raft_consensus.cc:2781] T 8cc62b872e2c4a6790d75ff4ecd560b0 P 6f9c1f7594aa43e5b1a4e3bd9dcdf7a1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 03:29:24.621745 17925 raft_consensus.cc:482] T 8cc62b872e2c4a6790d75ff4ecd560b0 P 6f9c1f7594aa43e5b1a4e3bd9dcdf7a1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 03:29:24.621847 17925 raft_consensus.cc:3037] T 8cc62b872e2c4a6790d75ff4ecd560b0 P 6f9c1f7594aa43e5b1a4e3bd9dcdf7a1 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:29:24.624092 17925 raft_consensus.cc:504] T 8cc62b872e2c4a6790d75ff4ecd560b0 P 6f9c1f7594aa43e5b1a4e3bd9dcdf7a1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "868802ad74164cc8b84e264576a9bb05" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 37585 } } peers { permanent_uuid: "6f9c1f7594aa43e5b1a4e3bd9dcdf7a1" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 45581 } } peers { permanent_uuid: "386dbc17f7974e99a6a7500ccc1d16f8" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 46513 } }
I20260810 03:29:24.624732 17925 leader_election.cc:288] T 8cc62b872e2c4a6790d75ff4ecd560b0 P 6f9c1f7594aa43e5b1a4e3bd9dcdf7a1 [CANDIDATE]: Term 1 election: Requested vote from peers 868802ad74164cc8b84e264576a9bb05 (127.27.161.1:37585), 386dbc17f7974e99a6a7500ccc1d16f8 (127.27.161.3:46513)
I20260810 03:29:24.625041 17401 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cc62b872e2c4a6790d75ff4ecd560b0" candidate_uuid: "6f9c1f7594aa43e5b1a4e3bd9dcdf7a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "868802ad74164cc8b84e264576a9bb05"
I20260810 03:29:24.625209 17401 raft_consensus.cc:3037] T 8cc62b872e2c4a6790d75ff4ecd560b0 P 868802ad74164cc8b84e264576a9bb05 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:29:24.625198 17808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cc62b872e2c4a6790d75ff4ecd560b0" candidate_uuid: "6f9c1f7594aa43e5b1a4e3bd9dcdf7a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "386dbc17f7974e99a6a7500ccc1d16f8"
I20260810 03:29:24.625379 17808 raft_consensus.cc:3037] T 8cc62b872e2c4a6790d75ff4ecd560b0 P 386dbc17f7974e99a6a7500ccc1d16f8 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:29:24.627476 17401 raft_consensus.cc:2454] T 8cc62b872e2c4a6790d75ff4ecd560b0 P 868802ad74164cc8b84e264576a9bb05 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6f9c1f7594aa43e5b1a4e3bd9dcdf7a1 in term 1.
I20260810 03:29:24.627578 17808 raft_consensus.cc:2454] T 8cc62b872e2c4a6790d75ff4ecd560b0 P 386dbc17f7974e99a6a7500ccc1d16f8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6f9c1f7594aa43e5b1a4e3bd9dcdf7a1 in term 1.
I20260810 03:29:24.627852 17499 leader_election.cc:302] T 8cc62b872e2c4a6790d75ff4ecd560b0 P 6f9c1f7594aa43e5b1a4e3bd9dcdf7a1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6f9c1f7594aa43e5b1a4e3bd9dcdf7a1, 868802ad74164cc8b84e264576a9bb05; no voters:
I20260810 03:29:24.628135 17925 raft_consensus.cc:2781] T 8cc62b872e2c4a6790d75ff4ecd560b0 P 6f9c1f7594aa43e5b1a4e3bd9dcdf7a1 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:29:24.628557 17925 raft_consensus.cc:686] T 8cc62b872e2c4a6790d75ff4ecd560b0 P 6f9c1f7594aa43e5b1a4e3bd9dcdf7a1 [term 1 LEADER]: Becoming Leader. State: Replica: 6f9c1f7594aa43e5b1a4e3bd9dcdf7a1, State: Running, Role: LEADER
I20260810 03:29:24.628988 17925 consensus_queue.cc:237] T 8cc62b872e2c4a6790d75ff4ecd560b0 P 6f9c1f7594aa43e5b1a4e3bd9dcdf7a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "868802ad74164cc8b84e264576a9bb05" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 37585 } } peers { permanent_uuid: "6f9c1f7594aa43e5b1a4e3bd9dcdf7a1" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 45581 } } peers { permanent_uuid: "386dbc17f7974e99a6a7500ccc1d16f8" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 46513 } }
I20260810 03:29:24.633255 17173 catalog_manager.cc:5495] T 8cc62b872e2c4a6790d75ff4ecd560b0 P 6f9c1f7594aa43e5b1a4e3bd9dcdf7a1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6f9c1f7594aa43e5b1a4e3bd9dcdf7a1 (127.27.161.2). New cstate: current_term: 1 leader_uuid: "6f9c1f7594aa43e5b1a4e3bd9dcdf7a1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "868802ad74164cc8b84e264576a9bb05" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 37585 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6f9c1f7594aa43e5b1a4e3bd9dcdf7a1" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 45581 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "386dbc17f7974e99a6a7500ccc1d16f8" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 46513 } health_report { overall_health: UNKNOWN } } }
W20260810 03:29:24.646523 17677 tablet.cc:2369] T 8cc62b872e2c4a6790d75ff4ecd560b0 P 6f9c1f7594aa43e5b1a4e3bd9dcdf7a1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 03:29:24.657848 17910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:29:24.652536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39168 (local address 127.27.161.2:45581)
0810 03:29:24.652731 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0810 03:29:24.652737 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:29:24.652756 (+ 19us) server_negotiation.cc:407] Connection header received
0810 03:29:24.652953 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:29:24.652957 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:29:24.653033 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:29:24.653171 (+ 138us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 03:29:24.653521 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:29:24.654271 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:29:24.655648 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:29:24.656106 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:29:24.656223 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:29:24.656709 (+ 486us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 03:29:24.657251 (+ 542us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 03:29:24.657400 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0810 03:29:24.657492 (+ 92us) server_negotiation.cc:299] Negotiation successful
0810 03:29:24.657574 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260810 03:29:24.666054 17401 raft_consensus.cc:1261] T 8cc62b872e2c4a6790d75ff4ecd560b0 P 868802ad74164cc8b84e264576a9bb05 [term 1 FOLLOWER]: Refusing update from remote peer 6f9c1f7594aa43e5b1a4e3bd9dcdf7a1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:29:24.666054 17808 raft_consensus.cc:1261] T 8cc62b872e2c4a6790d75ff4ecd560b0 P 386dbc17f7974e99a6a7500ccc1d16f8 [term 1 FOLLOWER]: Refusing update from remote peer 6f9c1f7594aa43e5b1a4e3bd9dcdf7a1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:29:24.667006 17925 consensus_queue.cc:1035] T 8cc62b872e2c4a6790d75ff4ecd560b0 P 6f9c1f7594aa43e5b1a4e3bd9dcdf7a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "386dbc17f7974e99a6a7500ccc1d16f8" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 46513 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:29:24.667287 17932 consensus_queue.cc:1035] T 8cc62b872e2c4a6790d75ff4ecd560b0 P 6f9c1f7594aa43e5b1a4e3bd9dcdf7a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "868802ad74164cc8b84e264576a9bb05" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 37585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260810 03:29:24.677444 17882 tablet.cc:2369] T 8cc62b872e2c4a6790d75ff4ecd560b0 P 386dbc17f7974e99a6a7500ccc1d16f8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 03:29:24.679123 17937 mvcc.cc:205] Tried to move back new op lower bound from 7316818184862543872 to 7316818184723877888. Current Snapshot: MvccSnapshot[applied={T|T < 7316818184862543872}]
I20260810 03:29:24.694409 17173 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58238:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 03:29:24.694662 17173 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58238:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 03:29:24.698240 17173 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 104edb053ce54ee1881fb2cbcace28cb: Sending DeleteTablet for 3 replicas of tablet 8cc62b872e2c4a6790d75ff4ecd560b0
I20260810 03:29:24.699306 17578 tablet_service.cc:1514] Processing DeleteTablet for tablet 8cc62b872e2c4a6790d75ff4ecd560b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:29:24 UTC) from {username='kudu', principal='kudu/127.27.161.62@KRBTEST.COM'} at 127.0.0.1:39166
I20260810 03:29:24.699352 17367 tablet_service.cc:1514] Processing DeleteTablet for tablet 8cc62b872e2c4a6790d75ff4ecd560b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:29:24 UTC) from {username='kudu', principal='kudu/127.27.161.62@KRBTEST.COM'} at 127.0.0.1:36946
I20260810 03:29:24.700177 17951 tablet_replica.cc:330] T 8cc62b872e2c4a6790d75ff4ecd560b0 P 6f9c1f7594aa43e5b1a4e3bd9dcdf7a1: stopping tablet replica
I20260810 03:29:24.700503 17778 tablet_service.cc:1514] Processing DeleteTablet for tablet 8cc62b872e2c4a6790d75ff4ecd560b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:29:24 UTC) from {username='kudu', principal='kudu/127.27.161.62@KRBTEST.COM'} at 127.0.0.1:36376
I20260810 03:29:24.700728 17951 raft_consensus.cc:2229] T 8cc62b872e2c4a6790d75ff4ecd560b0 P 6f9c1f7594aa43e5b1a4e3bd9dcdf7a1 [term 1 LEADER]: Raft consensus shutting down.
I20260810 03:29:24.701323 28292 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iBH5/build/asan/bin/kudu pid 17269
I20260810 03:29:24.701601 17951 raft_consensus.cc:2258] T 8cc62b872e2c4a6790d75ff4ecd560b0 P 6f9c1f7594aa43e5b1a4e3bd9dcdf7a1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:29:24.704067 17951 ts_tablet_manager.cc:1905] T 8cc62b872e2c4a6790d75ff4ecd560b0 P 6f9c1f7594aa43e5b1a4e3bd9dcdf7a1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:29:24.711992 17951 ts_tablet_manager.cc:1918] T 8cc62b872e2c4a6790d75ff4ecd560b0 P 6f9c1f7594aa43e5b1a4e3bd9dcdf7a1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 03:29:24.712265 17951 log.cc:1196] T 8cc62b872e2c4a6790d75ff4ecd560b0 P 6f9c1f7594aa43e5b1a4e3bd9dcdf7a1: Deleting WAL directory at /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/minicluster-data/ts-1/wal/wals/8cc62b872e2c4a6790d75ff4ecd560b0
I20260810 03:29:24.712896 17951 ts_tablet_manager.cc:1939] T 8cc62b872e2c4a6790d75ff4ecd560b0 P 6f9c1f7594aa43e5b1a4e3bd9dcdf7a1: Deleting consensus metadata
I20260810 03:29:24.714960 17142 catalog_manager.cc:4841] TS 6f9c1f7594aa43e5b1a4e3bd9dcdf7a1 (127.27.161.2:45581): tablet 8cc62b872e2c4a6790d75ff4ecd560b0 (table test-table [id=575497a7045e47538c60ac043e0364c1]) successfully deleted
I20260810 03:29:24.719295 17955 tablet_replica.cc:330] T 8cc62b872e2c4a6790d75ff4ecd560b0 P 386dbc17f7974e99a6a7500ccc1d16f8: stopping tablet replica
I20260810 03:29:24.719894 17955 raft_consensus.cc:2229] T 8cc62b872e2c4a6790d75ff4ecd560b0 P 386dbc17f7974e99a6a7500ccc1d16f8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 03:29:24.720537 17955 raft_consensus.cc:2258] T 8cc62b872e2c4a6790d75ff4ecd560b0 P 386dbc17f7974e99a6a7500ccc1d16f8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:29:24.721683 17954 tablet_replica.cc:330] T 8cc62b872e2c4a6790d75ff4ecd560b0 P 868802ad74164cc8b84e264576a9bb05: stopping tablet replica
I20260810 03:29:24.722158 17954 raft_consensus.cc:2229] T 8cc62b872e2c4a6790d75ff4ecd560b0 P 868802ad74164cc8b84e264576a9bb05 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 03:29:24.722715 17954 raft_consensus.cc:2258] T 8cc62b872e2c4a6790d75ff4ecd560b0 P 868802ad74164cc8b84e264576a9bb05 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:29:24.723161 17955 ts_tablet_manager.cc:1905] T 8cc62b872e2c4a6790d75ff4ecd560b0 P 386dbc17f7974e99a6a7500ccc1d16f8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:29:24.724774 17954 ts_tablet_manager.cc:1905] T 8cc62b872e2c4a6790d75ff4ecd560b0 P 868802ad74164cc8b84e264576a9bb05: Deleting tablet data with delete state TABLET_DATA_DELETED
Aug 10 03:29:24 dist-test-slave-vpvm krb5kdc[17099](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332562, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.161.1@KRBTEST.COM
I20260810 03:29:24.731565 17955 ts_tablet_manager.cc:1918] T 8cc62b872e2c4a6790d75ff4ecd560b0 P 386dbc17f7974e99a6a7500ccc1d16f8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 03:29:24.731594 17954 ts_tablet_manager.cc:1918] T 8cc62b872e2c4a6790d75ff4ecd560b0 P 868802ad74164cc8b84e264576a9bb05: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 03:29:24.731740 17955 log.cc:1196] T 8cc62b872e2c4a6790d75ff4ecd560b0 P 386dbc17f7974e99a6a7500ccc1d16f8: Deleting WAL directory at /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/minicluster-data/ts-2/wal/wals/8cc62b872e2c4a6790d75ff4ecd560b0
I20260810 03:29:24.731920 17954 log.cc:1196] T 8cc62b872e2c4a6790d75ff4ecd560b0 P 868802ad74164cc8b84e264576a9bb05: Deleting WAL directory at /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/minicluster-data/ts-0/wal/wals/8cc62b872e2c4a6790d75ff4ecd560b0
I20260810 03:29:24.732273 17955 ts_tablet_manager.cc:1939] T 8cc62b872e2c4a6790d75ff4ecd560b0 P 386dbc17f7974e99a6a7500ccc1d16f8: Deleting consensus metadata
I20260810 03:29:24.732579 17954 ts_tablet_manager.cc:1939] T 8cc62b872e2c4a6790d75ff4ecd560b0 P 868802ad74164cc8b84e264576a9bb05: Deleting consensus metadata
I20260810 03:29:24.734146 17145 catalog_manager.cc:4841] TS 386dbc17f7974e99a6a7500ccc1d16f8 (127.27.161.3:46513): tablet 8cc62b872e2c4a6790d75ff4ecd560b0 (table test-table [id=575497a7045e47538c60ac043e0364c1]) successfully deleted
I20260810 03:29:24.734599 17151 catalog_manager.cc:4841] TS 868802ad74164cc8b84e264576a9bb05 (127.27.161.1:37585): tablet 8cc62b872e2c4a6790d75ff4ecd560b0 (table test-table [id=575497a7045e47538c60ac043e0364c1]) successfully deleted
I20260810 03:29:24.737121 17905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:29:24.702606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36950 (local address 127.27.161.1:37585)
0810 03:29:24.702787 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0810 03:29:24.702793 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:29:24.718604 (+ 15811us) server_negotiation.cc:407] Connection header received
0810 03:29:24.718830 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:29:24.718836 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:29:24.718930 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:29:24.719032 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:29:24.721094 (+ 2062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:29:24.722297 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:29:24.724471 (+ 2174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:29:24.725084 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:29:24.725200 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:29:24.730575 (+ 5375us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:29:24.730589 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:29:24.730608 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:29:24.730656 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:29:24.734419 (+ 3763us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:29:24.735188 (+ 769us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:29:24.735195 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:29:24.735198 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:29:24.735320 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:29:24.735679 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:29:24.735682 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:29:24.735684 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:29:24.736022 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:29:24.736309 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0810 03:29:24.736587 (+ 278us) server_negotiation.cc:299] Negotiation successful
0810 03:29:24.736797 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260810 03:29:24.737818 17455 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:29:24.836966 28292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iBH5/build/asan/bin/kudu with pid 17269
I20260810 03:29:24.865741 28292 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iBH5/build/asan/bin/kudu pid 17473
Aug 10 03:29:24 dist-test-slave-vpvm krb5kdc[17099](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332562, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.161.2@KRBTEST.COM
I20260810 03:29:24.889695 17910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:29:24.867051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39178 (local address 127.27.161.2:45581)
0810 03:29:24.867340 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0810 03:29:24.867373 (+ 33us) server_negotiation.cc:399] Waiting for connection header
0810 03:29:24.867391 (+ 18us) server_negotiation.cc:407] Connection header received
0810 03:29:24.867478 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:29:24.867484 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:29:24.867581 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:29:24.867676 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:29:24.872289 (+ 4613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:29:24.873423 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:29:24.874805 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:29:24.875450 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:29:24.875585 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:29:24.881066 (+ 5481us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:29:24.881082 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:29:24.881102 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:29:24.881150 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:29:24.886162 (+ 5012us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:29:24.887059 (+ 897us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:29:24.887070 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:29:24.887073 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:29:24.887261 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:29:24.887781 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:29:24.887786 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:29:24.887789 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:29:24.888317 (+ 528us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:29:24.888708 (+ 391us) server_negotiation.cc:1035] Waiting for connection context
0810 03:29:24.889017 (+ 309us) server_negotiation.cc:299] Negotiation successful
0810 03:29:24.889282 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106}
I20260810 03:29:24.890626 17667 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:29:25.012580 28292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iBH5/build/asan/bin/kudu with pid 17473
I20260810 03:29:25.035846 28292 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iBH5/build/asan/bin/kudu pid 17684
Aug 10 03:29:25 dist-test-slave-vpvm krb5kdc[17099](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332562, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.161.3@KRBTEST.COM
I20260810 03:29:25.050668 17911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:29:25.036597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36392 (local address 127.27.161.3:46513)
0810 03:29:25.036781 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0810 03:29:25.036787 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:29:25.036807 (+ 20us) server_negotiation.cc:407] Connection header received
0810 03:29:25.036937 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:29:25.036942 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:29:25.037017 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:29:25.037102 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:29:25.038514 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:29:25.039241 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:29:25.040537 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:29:25.040925 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:29:25.041035 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:29:25.045441 (+ 4406us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:29:25.045455 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:29:25.045472 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:29:25.045504 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:29:25.047947 (+ 2443us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:29:25.048577 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:29:25.048585 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:29:25.048587 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:29:25.048714 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:29:25.049055 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:29:25.049059 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:29:25.049061 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:29:25.049410 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:29:25.049696 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0810 03:29:25.050179 (+ 483us) server_negotiation.cc:299] Negotiation successful
0810 03:29:25.050400 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260810 03:29:25.051365 17873 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:29:25.139496 28292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iBH5/build/asan/bin/kudu with pid 17684
I20260810 03:29:25.162544 28292 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iBH5/build/asan/bin/kudu pid 17119
I20260810 03:29:25.163664 17221 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:29:25.271891 28292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iBH5/build/asan/bin/kudu with pid 17119
I20260810 03:29:25.300032 28292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iBH5/build/asan/bin/kudu
/tmp/dist-test-task24iBH5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.161.62:35141
--webserver_interface=127.27.161.62
--webserver_port=35053
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.62.keytab
--principal=kudu/127.27.161.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.161.20:36555
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.161.62:35141
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:29:25.632838 17964 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:29:25.633152 17964 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:29:25.633282 17964 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:29:25.640564 17964 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:29:25.640681 17964 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:29:25.640758 17964 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:29:25.640830 17964 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:29:25.640893 17964 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:29:25.650823 17964 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.161.20:36555
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.27.161.62:35141
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.62.keytab
--principal=kudu/127.27.161.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.161.62:35141
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.161.62
--webserver_port=35053
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:29:25.652170 17964 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:29:25.653472 17964 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:29:25.678135 17973 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:29:25.678179 17974 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:29:25.678099 17976 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:29:25.679260 17964 server_base.cc:1034] running on GCE node
I20260810 03:29:25.680154 17964 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:29:25.681715 17964 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:29:25.682909 17964 hybrid_clock.cc:648] HybridClock initialized: now 1786332565682831 us; error 62 us; skew 500 ppm
I20260810 03:29:25.683307 17964 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:29:25 dist-test-slave-vpvm krb5kdc[17099](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332565, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.161.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:29:25.690253 17964 init.cc:377] Logged in from keytab as kudu/127.27.161.62@KRBTEST.COM (short username kudu)
I20260810 03:29:25.691898 17964 webserver.cc:460] Webserver started at http://127.27.161.62:35053/ using document root <none> and password file <none>
I20260810 03:29:25.692688 17964 fs_manager.cc:359] Metadata directory not provided
I20260810 03:29:25.692806 17964 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:29:25.698711 17964 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260810 03:29:25.701639 17985 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:29:25.703027 17964 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260810 03:29:25.703209 17964 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/minicluster-data/master-0/data,/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/minicluster-data/master-0/wal
uuid: "104edb053ce54ee1881fb2cbcace28cb"
format_stamp: "Formatted at 2026-08-10 03:29:22 on dist-test-slave-vpvm"
server_key: "aff4ca34f5f297d99297f8303e9c1a5b"
server_key_iv: "eab5ab9508a75442c91afc3bc3405470"
server_key_version: "encryptionkey@0"
I20260810 03:29:25.704315 17964 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:29:25.733856 17964 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:29:25.738425 17964 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:29:25.738710 17964 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:29:25.756507 17964 rpc_server.cc:307] RPC server started. Bound to: 127.27.161.62:35141
I20260810 03:29:25.756531 18061 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.161.62:35141 every 8 connection(s)
I20260810 03:29:25.758023 17964 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/minicluster-data/master-0/data/info.pb
I20260810 03:29:25.765487 18062 sys_catalog.cc:263] Verifying existing consensus state
I20260810 03:29:25.765524 28292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iBH5/build/asan/bin/kudu as pid 17964
I20260810 03:29:25.766356 28292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iBH5/build/asan/bin/kudu
/tmp/dist-test-task24iBH5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.161.1:37585
--local_ip_for_outbound_sockets=127.27.161.1
--tserver_master_addrs=127.27.161.62:35141
--webserver_port=37695
--webserver_interface=127.27.161.1
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.1.keytab
--principal=kudu/127.27.161.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.161.20:36555
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 03:29:25.768986 18062 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 104edb053ce54ee1881fb2cbcace28cb: Bootstrap starting.
I20260810 03:29:25.788550 18062 log.cc:824] T 00000000000000000000000000000000 P 104edb053ce54ee1881fb2cbcace28cb: Log is configured to *not* fsync() on all Append() calls
I20260810 03:29:25.801185 18062 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 104edb053ce54ee1881fb2cbcace28cb: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 03:29:25.801708 18062 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 104edb053ce54ee1881fb2cbcace28cb: Bootstrap complete.
I20260810 03:29:25.811211 18062 raft_consensus.cc:348] T 00000000000000000000000000000000 P 104edb053ce54ee1881fb2cbcace28cb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "104edb053ce54ee1881fb2cbcace28cb" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 35141 } }
I20260810 03:29:25.812052 18062 raft_consensus.cc:729] T 00000000000000000000000000000000 P 104edb053ce54ee1881fb2cbcace28cb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 104edb053ce54ee1881fb2cbcace28cb, State: Initialized, Role: FOLLOWER
I20260810 03:29:25.812660 18062 consensus_queue.cc:260] T 00000000000000000000000000000000 P 104edb053ce54ee1881fb2cbcace28cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "104edb053ce54ee1881fb2cbcace28cb" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 35141 } }
I20260810 03:29:25.812891 18062 raft_consensus.cc:388] T 00000000000000000000000000000000 P 104edb053ce54ee1881fb2cbcace28cb [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 03:29:25.813011 18062 raft_consensus.cc:482] T 00000000000000000000000000000000 P 104edb053ce54ee1881fb2cbcace28cb [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 03:29:25.813177 18062 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 104edb053ce54ee1881fb2cbcace28cb [term 1 FOLLOWER]: Advancing to term 2
I20260810 03:29:25.816963 18062 raft_consensus.cc:504] T 00000000000000000000000000000000 P 104edb053ce54ee1881fb2cbcace28cb [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "104edb053ce54ee1881fb2cbcace28cb" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 35141 } }
I20260810 03:29:25.817461 18062 leader_election.cc:302] T 00000000000000000000000000000000 P 104edb053ce54ee1881fb2cbcace28cb [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 104edb053ce54ee1881fb2cbcace28cb; no voters:
I20260810 03:29:25.820393 18062 leader_election.cc:288] T 00000000000000000000000000000000 P 104edb053ce54ee1881fb2cbcace28cb [CANDIDATE]: Term 2 election: Requested vote from peers
I20260810 03:29:25.820540 18068 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 104edb053ce54ee1881fb2cbcace28cb [term 2 FOLLOWER]: Leader election won for term 2
I20260810 03:29:25.822242 18068 raft_consensus.cc:686] T 00000000000000000000000000000000 P 104edb053ce54ee1881fb2cbcace28cb [term 2 LEADER]: Becoming Leader. State: Replica: 104edb053ce54ee1881fb2cbcace28cb, State: Running, Role: LEADER
I20260810 03:29:25.822764 18062 sys_catalog.cc:564] T 00000000000000000000000000000000 P 104edb053ce54ee1881fb2cbcace28cb [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:29:25.822871 18068 consensus_queue.cc:237] T 00000000000000000000000000000000 P 104edb053ce54ee1881fb2cbcace28cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "104edb053ce54ee1881fb2cbcace28cb" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 35141 } }
I20260810 03:29:25.825572 18071 sys_catalog.cc:455] T 00000000000000000000000000000000 P 104edb053ce54ee1881fb2cbcace28cb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 104edb053ce54ee1881fb2cbcace28cb. Latest consensus state: current_term: 2 leader_uuid: "104edb053ce54ee1881fb2cbcace28cb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "104edb053ce54ee1881fb2cbcace28cb" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 35141 } } }
I20260810 03:29:25.825637 18070 sys_catalog.cc:455] T 00000000000000000000000000000000 P 104edb053ce54ee1881fb2cbcace28cb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "104edb053ce54ee1881fb2cbcace28cb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "104edb053ce54ee1881fb2cbcace28cb" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 35141 } } }
I20260810 03:29:25.826041 18070 sys_catalog.cc:458] T 00000000000000000000000000000000 P 104edb053ce54ee1881fb2cbcace28cb [sys.catalog]: This master's current role is: LEADER
I20260810 03:29:25.826130 18071 sys_catalog.cc:458] T 00000000000000000000000000000000 P 104edb053ce54ee1881fb2cbcace28cb [sys.catalog]: This master's current role is: LEADER
I20260810 03:29:25.838587 18080 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 03:29:25.846907 18080 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 03:29:25.848436 18080 catalog_manager.cc:1259] Loaded cluster ID: 04b129a66593464187ede0ccb2dd4272
I20260810 03:29:25.848596 18080 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 03:29:25.851828 18080 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260810 03:29:25.869261 18080 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260810 03:29:25.877382 18080 catalog_manager.cc:1504] Loading token signing keys...
I20260810 03:29:25.878803 18080 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 104edb053ce54ee1881fb2cbcace28cb: Loaded TSK: 0
I20260810 03:29:25.879907 18080 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260810 03:29:26.141175 18066 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:29:26.141470 18066 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:29:26.141592 18066 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:29:26.149022 18066 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:29:26.149145 18066 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:29:26.149302 18066 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.161.1
I20260810 03:29:26.159277 18066 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.161.20:36555
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.1.keytab
--principal=kudu/127.27.161.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.161.1:37585
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.161.1
--webserver_port=37695
--tserver_master_addrs=127.27.161.62:35141
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.161.1
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:29:26.160677 18066 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:29:26.162071 18066 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:29:26.176751 18101 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:29:26.177196 18105 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:29:26.177273 18100 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:29:26.178066 18066 server_base.cc:1034] running on GCE node
I20260810 03:29:26.178675 18066 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:29:26.179701 18066 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:29:26.180903 18066 hybrid_clock.cc:648] HybridClock initialized: now 1786332566180842 us; error 55 us; skew 500 ppm
I20260810 03:29:26.181205 18066 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:29:26 dist-test-slave-vpvm krb5kdc[17099](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332566, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.161.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:29:26.184465 18066 init.cc:377] Logged in from keytab as kudu/127.27.161.1@KRBTEST.COM (short username kudu)
I20260810 03:29:26.186020 18066 webserver.cc:460] Webserver started at http://127.27.161.1:37695/ using document root <none> and password file <none>
I20260810 03:29:26.186699 18066 fs_manager.cc:359] Metadata directory not provided
I20260810 03:29:26.186836 18066 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:29:26.192183 18066 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.004s
I20260810 03:29:26.195086 18112 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:29:26.196444 18066 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260810 03:29:26.196594 18066 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/minicluster-data/ts-0/data,/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/minicluster-data/ts-0/wal
uuid: "868802ad74164cc8b84e264576a9bb05"
format_stamp: "Formatted at 2026-08-10 03:29:23 on dist-test-slave-vpvm"
server_key: "844fdd946a70ccfd1a823157e16e41cb"
server_key_iv: "c4bddeddb2a89fbf0690af36a364c10e"
server_key_version: "encryptionkey@0"
I20260810 03:29:26.197549 18066 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:29:26.210644 18066 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:29:26.214859 18066 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:29:26.215111 18066 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:29:26.216173 18066 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:29:26.217610 18066 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:29:26.217700 18066 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:29:26.217803 18066 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:29:26.217875 18066 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:29:26.249653 18066 rpc_server.cc:307] RPC server started. Bound to: 127.27.161.1:37585
I20260810 03:29:26.249709 18302 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.161.1:37585 every 8 connection(s)
I20260810 03:29:26.251648 18066 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/minicluster-data/ts-0/data/info.pb
I20260810 03:29:26.255996 28292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iBH5/build/asan/bin/kudu as pid 18066
I20260810 03:29:26.256778 28292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iBH5/build/asan/bin/kudu
/tmp/dist-test-task24iBH5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.161.2:45581
--local_ip_for_outbound_sockets=127.27.161.2
--tserver_master_addrs=127.27.161.62:35141
--webserver_port=34879
--webserver_interface=127.27.161.2
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.2.keytab
--principal=kudu/127.27.161.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.161.20:36555
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 03:29:26 dist-test-slave-vpvm krb5kdc[17099](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332566, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.161.1@KRBTEST.COM for kudu/127.27.161.62@KRBTEST.COM
I20260810 03:29:26.278478 18309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:29:26.255310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:35141 (local address 127.27.161.1:54159)
0810 03:29:26.257022 (+ 1712us) negotiation.cc:107] Waiting for socket to connect
0810 03:29:26.257075 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0810 03:29:26.258996 (+ 1921us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:29:26.260557 (+ 1561us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:29:26.260568 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:29:26.261266 (+ 698us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:29:26.262053 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:29:26.262083 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:29:26.263736 (+ 1653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:29:26.263741 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:29:26.265459 (+ 1718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:29:26.265475 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:29:26.266483 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:29:26.266491 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:29:26.266739 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:29:26.267439 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:29:26.267467 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:29:26.270047 (+ 2580us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:29:26.274247 (+ 4200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:29:26.274255 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:29:26.274267 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:29:26.274715 (+ 448us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:29:26.275260 (+ 545us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:29:26.275263 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:29:26.275265 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:29:26.275455 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:29:26.276381 (+ 926us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:29:26.276388 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:29:26.276707 (+ 319us) client_negotiation.cc:769] Sending connection context
0810 03:29:26.276944 (+ 237us) client_negotiation.cc:240] Negotiation successful
0810 03:29:26.277380 (+ 436us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1396,"thread_start_us":1166,"threads_started":1}
I20260810 03:29:26.278802 18308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:29:26.255088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.161.1:54159 (local address 127.27.161.62:35141)
0810 03:29:26.255996 (+ 908us) server_negotiation.cc:206] Beginning negotiation
0810 03:29:26.256013 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0810 03:29:26.259302 (+ 3289us) server_negotiation.cc:407] Connection header received
0810 03:29:26.259691 (+ 389us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:29:26.259704 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:29:26.260040 (+ 336us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:29:26.260324 (+ 284us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:29:26.262300 (+ 1976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:29:26.263545 (+ 1245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:29:26.265583 (+ 2038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:29:26.266251 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:29:26.266382 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:29:26.270352 (+ 3970us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:29:26.270363 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:29:26.270382 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:29:26.270437 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:29:26.274081 (+ 3644us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:29:26.274878 (+ 797us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:29:26.274887 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:29:26.274894 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:29:26.275113 (+ 219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:29:26.275589 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:29:26.275594 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:29:26.275597 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:29:26.276213 (+ 616us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:29:26.276407 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0810 03:29:26.277266 (+ 859us) server_negotiation.cc:299] Negotiation successful
0810 03:29:26.277669 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":565,"thread_start_us":4410,"threads_started":1}
I20260810 03:29:26.281450 18304 heartbeater.cc:347] Connected to a master server at 127.27.161.62:35141
I20260810 03:29:26.281773 18304 heartbeater.cc:464] Registering TS with master...
I20260810 03:29:26.282666 18304 heartbeater.cc:511] Master 127.27.161.62:35141 requested a full tablet report, sending...
I20260810 03:29:26.284968 18009 ts_manager.cc:194] Registered new tserver with Master: 868802ad74164cc8b84e264576a9bb05 (127.27.161.1:37585)
I20260810 03:29:26.286516 18009 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.161.1@KRBTEST.COM'} at 127.27.161.1:54159
W20260810 03:29:26.597009 18310 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:29:26.597286 18310 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:29:26.597407 18310 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:29:26.605068 18310 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:29:26.605178 18310 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:29:26.605325 18310 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.161.2
I20260810 03:29:26.615200 18310 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.161.20:36555
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.2.keytab
--principal=kudu/127.27.161.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.161.2:45581
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.161.2
--webserver_port=34879
--tserver_master_addrs=127.27.161.62:35141
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.161.2
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:29:26.616510 18310 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:29:26.617825 18310 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:29:26.631004 18318 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:29:26.631054 18323 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:29:26.631085 18320 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:29:26.631916 18310 server_base.cc:1034] running on GCE node
I20260810 03:29:26.632526 18310 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:29:26.633436 18310 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:29:26.634639 18310 hybrid_clock.cc:648] HybridClock initialized: now 1786332566634582 us; error 42 us; skew 500 ppm
I20260810 03:29:26.634943 18310 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:29:26 dist-test-slave-vpvm krb5kdc[17099](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332566, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.161.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:29:26.638020 18310 init.cc:377] Logged in from keytab as kudu/127.27.161.2@KRBTEST.COM (short username kudu)
I20260810 03:29:26.639636 18310 webserver.cc:460] Webserver started at http://127.27.161.2:34879/ using document root <none> and password file <none>
I20260810 03:29:26.640309 18310 fs_manager.cc:359] Metadata directory not provided
I20260810 03:29:26.640427 18310 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:29:26.645751 18310 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260810 03:29:26.648379 18333 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:29:26.649665 18310 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 03:29:26.649804 18310 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/minicluster-data/ts-1/data,/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/minicluster-data/ts-1/wal
uuid: "6f9c1f7594aa43e5b1a4e3bd9dcdf7a1"
format_stamp: "Formatted at 2026-08-10 03:29:23 on dist-test-slave-vpvm"
server_key: "8ae896ece668e6bb6925e22796815542"
server_key_iv: "ff012c096deb7a32414389673d403323"
server_key_version: "encryptionkey@0"
I20260810 03:29:26.650758 18310 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:29:26.668424 18310 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:29:26.672165 18310 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:29:26.672374 18310 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:29:26.673285 18310 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:29:26.674665 18310 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:29:26.674754 18310 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:29:26.674837 18310 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:29:26.674891 18310 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:29:26.705802 18310 rpc_server.cc:307] RPC server started. Bound to: 127.27.161.2:45581
I20260810 03:29:26.705853 18494 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.161.2:45581 every 8 connection(s)
I20260810 03:29:26.707240 18310 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/minicluster-data/ts-1/data/info.pb
I20260810 03:29:26.711434 28292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iBH5/build/asan/bin/kudu as pid 18310
I20260810 03:29:26.712081 28292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iBH5/build/asan/bin/kudu
/tmp/dist-test-task24iBH5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.161.3:46513
--local_ip_for_outbound_sockets=127.27.161.3
--tserver_master_addrs=127.27.161.62:35141
--webserver_port=33479
--webserver_interface=127.27.161.3
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.3.keytab
--principal=kudu/127.27.161.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.161.20:36555
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 03:29:26 dist-test-slave-vpvm krb5kdc[17099](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332566, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.161.2@KRBTEST.COM for kudu/127.27.161.62@KRBTEST.COM
I20260810 03:29:26.727733 18308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:29:26.709850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.161.2:44057 (local address 127.27.161.62:35141)
0810 03:29:26.710053 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0810 03:29:26.710057 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:29:26.711285 (+ 1228us) server_negotiation.cc:407] Connection header received
0810 03:29:26.713214 (+ 1929us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:29:26.713220 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:29:26.713307 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:29:26.713396 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:29:26.715106 (+ 1710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:29:26.715806 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:29:26.717383 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:29:26.717796 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:29:26.717909 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:29:26.721596 (+ 3687us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:29:26.721602 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:29:26.721605 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:29:26.721644 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:29:26.724321 (+ 2677us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:29:26.725211 (+ 890us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:29:26.725218 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:29:26.725221 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:29:26.725377 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:29:26.725803 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:29:26.725808 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:29:26.725812 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:29:26.726194 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:29:26.726334 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0810 03:29:26.727140 (+ 806us) server_negotiation.cc:299] Negotiation successful
0810 03:29:26.727415 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260810 03:29:26.728582 18499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:29:26.710409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:35141 (local address 127.27.161.2:44057)
0810 03:29:26.711122 (+ 713us) negotiation.cc:107] Waiting for socket to connect
0810 03:29:26.711163 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0810 03:29:26.712918 (+ 1755us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:29:26.713670 (+ 752us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:29:26.713679 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:29:26.714315 (+ 636us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:29:26.714920 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:29:26.714942 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:29:26.715970 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:29:26.715974 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:29:26.717219 (+ 1245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:29:26.717234 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:29:26.718065 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:29:26.718075 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:29:26.718287 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:29:26.718927 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:29:26.718954 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:29:26.721351 (+ 2397us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:29:26.724474 (+ 3123us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:29:26.724482 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:29:26.724497 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:29:26.725051 (+ 554us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:29:26.725480 (+ 429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:29:26.725483 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:29:26.725486 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:29:26.725684 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:29:26.726314 (+ 630us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:29:26.726320 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:29:26.726707 (+ 387us) client_negotiation.cc:769] Sending connection context
0810 03:29:26.726988 (+ 281us) client_negotiation.cc:240] Negotiation successful
0810 03:29:26.727431 (+ 443us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":278,"threads_started":1}
I20260810 03:29:26.730021 18496 heartbeater.cc:347] Connected to a master server at 127.27.161.62:35141
I20260810 03:29:26.730321 18496 heartbeater.cc:464] Registering TS with master...
I20260810 03:29:26.731097 18496 heartbeater.cc:511] Master 127.27.161.62:35141 requested a full tablet report, sending...
I20260810 03:29:26.732679 18009 ts_manager.cc:194] Registered new tserver with Master: 6f9c1f7594aa43e5b1a4e3bd9dcdf7a1 (127.27.161.2:45581)
I20260810 03:29:26.733533 18009 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.161.2@KRBTEST.COM'} at 127.27.161.2:44057
W20260810 03:29:27.051015 18500 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:29:27.051311 18500 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:29:27.051453 18500 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:29:27.058972 18500 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:29:27.059098 18500 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:29:27.059259 18500 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.161.3
I20260810 03:29:27.069298 18500 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.161.20:36555
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.3.keytab
--principal=kudu/127.27.161.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.161.3:46513
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.161.3
--webserver_port=33479
--tserver_master_addrs=127.27.161.62:35141
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.161.3
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:29:27.070672 18500 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:29:27.072005 18500 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:29:27.085733 18510 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:29:27.085996 18513 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:29:27.086494 18511 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:29:27.086979 18500 server_base.cc:1034] running on GCE node
I20260810 03:29:27.087586 18500 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:29:27.088526 18500 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:29:27.089716 18500 hybrid_clock.cc:648] HybridClock initialized: now 1786332567089658 us; error 54 us; skew 500 ppm
I20260810 03:29:27.090056 18500 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:29:27 dist-test-slave-vpvm krb5kdc[17099](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332567, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.161.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:29:27.093215 18500 init.cc:377] Logged in from keytab as kudu/127.27.161.3@KRBTEST.COM (short username kudu)
I20260810 03:29:27.094762 18500 webserver.cc:460] Webserver started at http://127.27.161.3:33479/ using document root <none> and password file <none>
I20260810 03:29:27.095486 18500 fs_manager.cc:359] Metadata directory not provided
I20260810 03:29:27.095625 18500 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:29:27.100957 18500 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.003s
I20260810 03:29:27.103818 18528 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:29:27.105211 18500 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260810 03:29:27.105368 18500 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/minicluster-data/ts-2/data,/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/minicluster-data/ts-2/wal
uuid: "386dbc17f7974e99a6a7500ccc1d16f8"
format_stamp: "Formatted at 2026-08-10 03:29:24 on dist-test-slave-vpvm"
server_key: "c10c20a349ff82746b8dc4eca2a2671b"
server_key_iv: "f0e46891c61ae3b300c0791780f9daeb"
server_key_version: "encryptionkey@0"
I20260810 03:29:27.106416 18500 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:29:27.124960 18500 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:29:27.128825 18500 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:29:27.129069 18500 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:29:27.130052 18500 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:29:27.131350 18500 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:29:27.131443 18500 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:29:27.131552 18500 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:29:27.131636 18500 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:29:27.159921 18500 rpc_server.cc:307] RPC server started. Bound to: 127.27.161.3:46513
I20260810 03:29:27.159970 18699 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.161.3:46513 every 8 connection(s)
I20260810 03:29:27.161342 18500 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/minicluster-data/ts-2/data/info.pb
I20260810 03:29:27.163152 28292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iBH5/build/asan/bin/kudu as pid 18500
Aug 10 03:29:27 dist-test-slave-vpvm krb5kdc[17099](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332567, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.161.3@KRBTEST.COM for kudu/127.27.161.62@KRBTEST.COM
I20260810 03:29:27.179908 18705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:29:27.163995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58246 (local address 127.27.161.62:35141)
0810 03:29:27.166891 (+ 2896us) server_negotiation.cc:206] Beginning negotiation
0810 03:29:27.166895 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:29:27.166909 (+ 14us) server_negotiation.cc:407] Connection header received
0810 03:29:27.166961 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:29:27.166964 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:29:27.167023 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:29:27.167116 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:29:27.168709 (+ 1593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:29:27.169481 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:29:27.170896 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:29:27.171299 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:29:27.171406 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:29:27.174078 (+ 2672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:29:27.174083 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:29:27.174086 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:29:27.174120 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:29:27.177217 (+ 3097us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:29:27.177868 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:29:27.177873 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:29:27.177875 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:29:27.178041 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:29:27.178426 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:29:27.178432 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:29:27.178435 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:29:27.178784 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:29:27.179029 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0810 03:29:27.179318 (+ 289us) server_negotiation.cc:299] Negotiation successful
0810 03:29:27.179505 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":20,"server-negotiator.queue_time_us":2804,"thread_start_us":2673,"threads_started":1}
I20260810 03:29:27.181721 18308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:29:27.163915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.161.3:41095 (local address 127.27.161.62:35141)
0810 03:29:27.164143 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0810 03:29:27.164149 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:29:27.165659 (+ 1510us) server_negotiation.cc:407] Connection header received
0810 03:29:27.167695 (+ 2036us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:29:27.167700 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:29:27.167771 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:29:27.167852 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:29:27.169345 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:29:27.170109 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:29:27.171731 (+ 1622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:29:27.172150 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:29:27.172253 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:29:27.175975 (+ 3722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:29:27.175983 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:29:27.175988 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:29:27.176042 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:29:27.178521 (+ 2479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:29:27.179333 (+ 812us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:29:27.179338 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:29:27.179341 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:29:27.179494 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:29:27.179971 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:29:27.179977 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:29:27.179979 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:29:27.180300 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:29:27.180407 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0810 03:29:27.181259 (+ 852us) server_negotiation.cc:299] Negotiation successful
0810 03:29:27.181411 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260810 03:29:27.182878 18707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:29:27.164505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:35141 (local address 127.27.161.3:41095)
0810 03:29:27.165464 (+ 959us) negotiation.cc:107] Waiting for socket to connect
0810 03:29:27.165528 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0810 03:29:27.167428 (+ 1900us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:29:27.168099 (+ 671us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:29:27.168107 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:29:27.168606 (+ 499us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:29:27.169174 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:29:27.169192 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:29:27.170261 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:29:27.170264 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:29:27.171596 (+ 1332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:29:27.171609 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:29:27.172363 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:29:27.172370 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:29:27.172589 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:29:27.173165 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:29:27.173187 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:29:27.175725 (+ 2538us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:29:27.178649 (+ 2924us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:29:27.178658 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:29:27.178676 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:29:27.179214 (+ 538us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:29:27.179611 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:29:27.179616 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:29:27.179620 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:29:27.179857 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:29:27.180437 (+ 580us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:29:27.180445 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:29:27.180849 (+ 404us) client_negotiation.cc:769] Sending connection context
0810 03:29:27.181159 (+ 310us) client_negotiation.cc:240] Negotiation successful
0810 03:29:27.181631 (+ 472us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":621,"thread_start_us":357,"threads_started":1}
I20260810 03:29:27.184144 18701 heartbeater.cc:347] Connected to a master server at 127.27.161.62:35141
I20260810 03:29:27.184422 18701 heartbeater.cc:464] Registering TS with master...
I20260810 03:29:27.185133 18701 heartbeater.cc:511] Master 127.27.161.62:35141 requested a full tablet report, sending...
I20260810 03:29:27.186442 18009 ts_manager.cc:194] Registered new tserver with Master: 386dbc17f7974e99a6a7500ccc1d16f8 (127.27.161.3:46513)
I20260810 03:29:27.187222 18009 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.161.3@KRBTEST.COM'} at 127.27.161.3:41095
I20260810 03:29:27.194562 28292 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 03:29:27.200980 18308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:29:27.195428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58250 (local address 127.27.161.62:35141)
0810 03:29:27.195587 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0810 03:29:27.195591 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:29:27.196059 (+ 468us) server_negotiation.cc:407] Connection header received
0810 03:29:27.196248 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:29:27.196251 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:29:27.196376 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:29:27.196486 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 03:29:27.196849 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:29:27.197555 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:29:27.199182 (+ 1627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:29:27.199589 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:29:27.199687 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:29:27.200089 (+ 402us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 03:29:27.200577 (+ 488us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 03:29:27.200688 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0810 03:29:27.200725 (+ 37us) server_negotiation.cc:299] Negotiation successful
0810 03:29:27.200788 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260810 03:29:27.201900 18009 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:58250:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 03:29:27.209168 18009 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 03:29:27.230791 18716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:29:27.220424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.3:46513 (local address 127.0.0.1:36398)
0810 03:29:27.221999 (+ 1575us) negotiation.cc:107] Waiting for socket to connect
0810 03:29:27.222015 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 03:29:27.222139 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:29:27.222880 (+ 741us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:29:27.222887 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:29:27.222917 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:29:27.223384 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:29:27.223402 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:29:27.225358 (+ 1956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:29:27.225363 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:29:27.228101 (+ 2738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:29:27.228115 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:29:27.229780 (+ 1665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:29:27.229787 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:29:27.230002 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:29:27.230061 (+ 59us) client_negotiation.cc:769] Sending connection context
0810 03:29:27.230199 (+ 138us) client_negotiation.cc:240] Negotiation successful
0810 03:29:27.230382 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1441,"mutex_wait_us":3,"thread_start_us":1298,"threads_started":1}
I20260810 03:29:27.230793 18718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:29:27.220916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36398 (local address 127.27.161.3:46513)
0810 03:29:27.221897 (+ 981us) server_negotiation.cc:206] Beginning negotiation
0810 03:29:27.221904 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:29:27.222255 (+ 351us) server_negotiation.cc:407] Connection header received
0810 03:29:27.222358 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:29:27.222366 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:29:27.222657 (+ 291us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:29:27.222847 (+ 190us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:29:27.223659 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:29:27.225195 (+ 1536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:29:27.228251 (+ 3056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:29:27.229570 (+ 1319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:29:27.229688 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:29:27.229928 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0810 03:29:27.230263 (+ 335us) server_negotiation.cc:299] Negotiation successful
0810 03:29:27.230435 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":809,"thread_start_us":630,"threads_started":1}
I20260810 03:29:27.231657 18714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:29:27.220386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.2:45581 (local address 127.0.0.1:39190)
0810 03:29:27.221227 (+ 841us) negotiation.cc:107] Waiting for socket to connect
0810 03:29:27.221276 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0810 03:29:27.221439 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:29:27.224557 (+ 3118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:29:27.224562 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:29:27.224587 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:29:27.224927 (+ 340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:29:27.224938 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:29:27.226938 (+ 2000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:29:27.226942 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:29:27.228786 (+ 1844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:29:27.228806 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:29:27.230942 (+ 2136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:29:27.230949 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:29:27.231144 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:29:27.231176 (+ 32us) client_negotiation.cc:769] Sending connection context
0810 03:29:27.231238 (+ 62us) client_negotiation.cc:240] Negotiation successful
0810 03:29:27.231370 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":603,"mutex_wait_us":1,"thread_start_us":465,"threads_started":1}
I20260810 03:29:27.232563 18722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:29:27.222787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36956 (local address 127.27.161.1:37585)
0810 03:29:27.224664 (+ 1877us) server_negotiation.cc:206] Beginning negotiation
0810 03:29:27.224672 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 03:29:27.224699 (+ 27us) server_negotiation.cc:407] Connection header received
0810 03:29:27.224793 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:29:27.224801 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:29:27.225011 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:29:27.225500 (+ 489us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:29:27.226141 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:29:27.227667 (+ 1526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:29:27.230093 (+ 2426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:29:27.231541 (+ 1448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:29:27.231685 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:29:27.231864 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0810 03:29:27.232123 (+ 259us) server_negotiation.cc:299] Negotiation successful
0810 03:29:27.232272 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1718,"thread_start_us":1578,"threads_started":1}
I20260810 03:29:27.233083 18715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:29:27.220477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39190 (local address 127.27.161.2:45581)
0810 03:29:27.223972 (+ 3495us) server_negotiation.cc:206] Beginning negotiation
0810 03:29:27.223997 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0810 03:29:27.224030 (+ 33us) server_negotiation.cc:407] Connection header received
0810 03:29:27.224108 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:29:27.224113 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:29:27.224348 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:29:27.225114 (+ 766us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:29:27.225398 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:29:27.226809 (+ 1411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:29:27.228989 (+ 2180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:29:27.230698 (+ 1709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:29:27.232375 (+ 1677us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:29:27.232561 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0810 03:29:27.232626 (+ 65us) server_negotiation.cc:299] Negotiation successful
0810 03:29:27.232793 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3337,"thread_start_us":3198,"threads_started":1}
I20260810 03:29:27.234465 18721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:29:27.222687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.1:37585 (local address 127.0.0.1:36956)
0810 03:29:27.223711 (+ 1024us) negotiation.cc:107] Waiting for socket to connect
0810 03:29:27.223727 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 03:29:27.223830 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:29:27.225225 (+ 1395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:29:27.225230 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:29:27.225255 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:29:27.225884 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:29:27.225904 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:29:27.227803 (+ 1899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:29:27.227809 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:29:27.229895 (+ 2086us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:29:27.229911 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:29:27.231774 (+ 1863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:29:27.231781 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:29:27.231936 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:29:27.231969 (+ 33us) client_negotiation.cc:769] Sending connection context
0810 03:29:27.232055 (+ 86us) client_negotiation.cc:240] Negotiation successful
0810 03:29:27.234112 (+ 2057us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":865,"mutex_wait_us":23,"thread_start_us":713,"threads_started":1}
I20260810 03:29:27.235073 18609 tablet_service.cc:1467] Processing CreateTablet for tablet 8ccce1367a244b5f90bb820cfa222d9f (DEFAULT_TABLE table=test-table [id=feffebde7395446b9d00f5a22ddd2b85]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:29:27.236546 18609 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ccce1367a244b5f90bb820cfa222d9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:29:27.236227 18401 tablet_service.cc:1467] Processing CreateTablet for tablet 8ccce1367a244b5f90bb820cfa222d9f (DEFAULT_TABLE table=test-table [id=feffebde7395446b9d00f5a22ddd2b85]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:29:27.237293 18401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ccce1367a244b5f90bb820cfa222d9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:29:27.238003 18198 tablet_service.cc:1467] Processing CreateTablet for tablet 8ccce1367a244b5f90bb820cfa222d9f (DEFAULT_TABLE table=test-table [id=feffebde7395446b9d00f5a22ddd2b85]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:29:27.239357 18198 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ccce1367a244b5f90bb820cfa222d9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:29:27.250458 18723 tablet_bootstrap.cc:492] T 8ccce1367a244b5f90bb820cfa222d9f P 6f9c1f7594aa43e5b1a4e3bd9dcdf7a1: Bootstrap starting.
I20260810 03:29:27.254144 18723 tablet_bootstrap.cc:654] T 8ccce1367a244b5f90bb820cfa222d9f P 6f9c1f7594aa43e5b1a4e3bd9dcdf7a1: Neither blocks nor log segments found. Creating new log.
I20260810 03:29:27.255744 18723 log.cc:824] T 8ccce1367a244b5f90bb820cfa222d9f P 6f9c1f7594aa43e5b1a4e3bd9dcdf7a1: Log is configured to *not* fsync() on all Append() calls
I20260810 03:29:27.260881 18726 tablet_bootstrap.cc:492] T 8ccce1367a244b5f90bb820cfa222d9f P 868802ad74164cc8b84e264576a9bb05: Bootstrap starting.
I20260810 03:29:27.263751 18724 tablet_bootstrap.cc:492] T 8ccce1367a244b5f90bb820cfa222d9f P 386dbc17f7974e99a6a7500ccc1d16f8: Bootstrap starting.
I20260810 03:29:27.264926 18726 tablet_bootstrap.cc:654] T 8ccce1367a244b5f90bb820cfa222d9f P 868802ad74164cc8b84e264576a9bb05: Neither blocks nor log segments found. Creating new log.
I20260810 03:29:27.266278 18726 log.cc:824] T 8ccce1367a244b5f90bb820cfa222d9f P 868802ad74164cc8b84e264576a9bb05: Log is configured to *not* fsync() on all Append() calls
I20260810 03:29:27.267982 18723 tablet_bootstrap.cc:492] T 8ccce1367a244b5f90bb820cfa222d9f P 6f9c1f7594aa43e5b1a4e3bd9dcdf7a1: No bootstrap required, opened a new log
I20260810 03:29:27.268143 18724 tablet_bootstrap.cc:654] T 8ccce1367a244b5f90bb820cfa222d9f P 386dbc17f7974e99a6a7500ccc1d16f8: Neither blocks nor log segments found. Creating new log.
I20260810 03:29:27.268307 18723 ts_tablet_manager.cc:1397] T 8ccce1367a244b5f90bb820cfa222d9f P 6f9c1f7594aa43e5b1a4e3bd9dcdf7a1: Time spent bootstrapping tablet: real 0.018s user 0.004s sys 0.007s
I20260810 03:29:27.268785 18726 tablet_bootstrap.cc:492] T 8ccce1367a244b5f90bb820cfa222d9f P 868802ad74164cc8b84e264576a9bb05: No bootstrap required, opened a new log
I20260810 03:29:27.269017 18726 ts_tablet_manager.cc:1397] T 8ccce1367a244b5f90bb820cfa222d9f P 868802ad74164cc8b84e264576a9bb05: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260810 03:29:27.269398 18724 log.cc:824] T 8ccce1367a244b5f90bb820cfa222d9f P 386dbc17f7974e99a6a7500ccc1d16f8: Log is configured to *not* fsync() on all Append() calls
I20260810 03:29:27.272414 18724 tablet_bootstrap.cc:492] T 8ccce1367a244b5f90bb820cfa222d9f P 386dbc17f7974e99a6a7500ccc1d16f8: No bootstrap required, opened a new log
I20260810 03:29:27.272688 18724 ts_tablet_manager.cc:1397] T 8ccce1367a244b5f90bb820cfa222d9f P 386dbc17f7974e99a6a7500ccc1d16f8: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260810 03:29:27.276654 18726 raft_consensus.cc:348] T 8ccce1367a244b5f90bb820cfa222d9f P 868802ad74164cc8b84e264576a9bb05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f9c1f7594aa43e5b1a4e3bd9dcdf7a1" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 45581 } } peers { permanent_uuid: "386dbc17f7974e99a6a7500ccc1d16f8" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 46513 } } peers { permanent_uuid: "868802ad74164cc8b84e264576a9bb05" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 37585 } }
I20260810 03:29:27.276996 18726 raft_consensus.cc:374] T 8ccce1367a244b5f90bb820cfa222d9f P 868802ad74164cc8b84e264576a9bb05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:29:27.277122 18726 raft_consensus.cc:729] T 8ccce1367a244b5f90bb820cfa222d9f P 868802ad74164cc8b84e264576a9bb05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 868802ad74164cc8b84e264576a9bb05, State: Initialized, Role: FOLLOWER
I20260810 03:29:27.277688 18726 consensus_queue.cc:260] T 8ccce1367a244b5f90bb820cfa222d9f P 868802ad74164cc8b84e264576a9bb05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f9c1f7594aa43e5b1a4e3bd9dcdf7a1" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 45581 } } peers { permanent_uuid: "386dbc17f7974e99a6a7500ccc1d16f8" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 46513 } } peers { permanent_uuid: "868802ad74164cc8b84e264576a9bb05" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 37585 } }
I20260810 03:29:27.278800 18304 heartbeater.cc:503] Master 127.27.161.62:35141 was elected leader, sending a full tablet report...
I20260810 03:29:27.279277 18726 ts_tablet_manager.cc:1428] T 8ccce1367a244b5f90bb820cfa222d9f P 868802ad74164cc8b84e264576a9bb05: Time spent starting tablet: real 0.010s user 0.011s sys 0.000s
I20260810 03:29:27.280580 18723 raft_consensus.cc:348] T 8ccce1367a244b5f90bb820cfa222d9f P 6f9c1f7594aa43e5b1a4e3bd9dcdf7a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f9c1f7594aa43e5b1a4e3bd9dcdf7a1" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 45581 } } peers { permanent_uuid: "386dbc17f7974e99a6a7500ccc1d16f8" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 46513 } } peers { permanent_uuid: "868802ad74164cc8b84e264576a9bb05" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 37585 } }
I20260810 03:29:27.281010 18723 raft_consensus.cc:374] T 8ccce1367a244b5f90bb820cfa222d9f P 6f9c1f7594aa43e5b1a4e3bd9dcdf7a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:29:27.281128 18723 raft_consensus.cc:729] T 8ccce1367a244b5f90bb820cfa222d9f P 6f9c1f7594aa43e5b1a4e3bd9dcdf7a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f9c1f7594aa43e5b1a4e3bd9dcdf7a1, State: Initialized, Role: FOLLOWER
I20260810 03:29:27.281775 18723 consensus_queue.cc:260] T 8ccce1367a244b5f90bb820cfa222d9f P 6f9c1f7594aa43e5b1a4e3bd9dcdf7a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f9c1f7594aa43e5b1a4e3bd9dcdf7a1" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 45581 } } peers { permanent_uuid: "386dbc17f7974e99a6a7500ccc1d16f8" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 46513 } } peers { permanent_uuid: "868802ad74164cc8b84e264576a9bb05" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 37585 } }
I20260810 03:29:27.282646 18724 raft_consensus.cc:348] T 8ccce1367a244b5f90bb820cfa222d9f P 386dbc17f7974e99a6a7500ccc1d16f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f9c1f7594aa43e5b1a4e3bd9dcdf7a1" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 45581 } } peers { permanent_uuid: "386dbc17f7974e99a6a7500ccc1d16f8" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 46513 } } peers { permanent_uuid: "868802ad74164cc8b84e264576a9bb05" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 37585 } }
I20260810 03:29:27.283059 18496 heartbeater.cc:503] Master 127.27.161.62:35141 was elected leader, sending a full tablet report...
I20260810 03:29:27.283067 18724 raft_consensus.cc:374] T 8ccce1367a244b5f90bb820cfa222d9f P 386dbc17f7974e99a6a7500ccc1d16f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:29:27.283166 18724 raft_consensus.cc:729] T 8ccce1367a244b5f90bb820cfa222d9f P 386dbc17f7974e99a6a7500ccc1d16f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 386dbc17f7974e99a6a7500ccc1d16f8, State: Initialized, Role: FOLLOWER
I20260810 03:29:27.283748 18723 ts_tablet_manager.cc:1428] T 8ccce1367a244b5f90bb820cfa222d9f P 6f9c1f7594aa43e5b1a4e3bd9dcdf7a1: Time spent starting tablet: real 0.015s user 0.012s sys 0.002s
I20260810 03:29:27.283789 18724 consensus_queue.cc:260] T 8ccce1367a244b5f90bb820cfa222d9f P 386dbc17f7974e99a6a7500ccc1d16f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f9c1f7594aa43e5b1a4e3bd9dcdf7a1" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 45581 } } peers { permanent_uuid: "386dbc17f7974e99a6a7500ccc1d16f8" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 46513 } } peers { permanent_uuid: "868802ad74164cc8b84e264576a9bb05" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 37585 } }
I20260810 03:29:27.285554 18701 heartbeater.cc:503] Master 127.27.161.62:35141 was elected leader, sending a full tablet report...
I20260810 03:29:27.286194 18724 ts_tablet_manager.cc:1428] T 8ccce1367a244b5f90bb820cfa222d9f P 386dbc17f7974e99a6a7500ccc1d16f8: Time spent starting tablet: real 0.013s user 0.015s sys 0.000s
W20260810 03:29:27.413163 18704 tablet.cc:2369] T 8ccce1367a244b5f90bb820cfa222d9f P 386dbc17f7974e99a6a7500ccc1d16f8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 03:29:27.459200 18497 tablet.cc:2369] T 8ccce1367a244b5f90bb820cfa222d9f P 6f9c1f7594aa43e5b1a4e3bd9dcdf7a1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 03:29:27.467260 18733 raft_consensus.cc:482] T 8ccce1367a244b5f90bb820cfa222d9f P 6f9c1f7594aa43e5b1a4e3bd9dcdf7a1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 03:29:27.467576 18733 raft_consensus.cc:504] T 8ccce1367a244b5f90bb820cfa222d9f P 6f9c1f7594aa43e5b1a4e3bd9dcdf7a1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f9c1f7594aa43e5b1a4e3bd9dcdf7a1" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 45581 } } peers { permanent_uuid: "386dbc17f7974e99a6a7500ccc1d16f8" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 46513 } } peers { permanent_uuid: "868802ad74164cc8b84e264576a9bb05" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 37585 } }
I20260810 03:29:27.469218 18733 leader_election.cc:288] T 8ccce1367a244b5f90bb820cfa222d9f P 6f9c1f7594aa43e5b1a4e3bd9dcdf7a1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 386dbc17f7974e99a6a7500ccc1d16f8 (127.27.161.3:46513), 868802ad74164cc8b84e264576a9bb05 (127.27.161.1:37585)
I20260810 03:29:27.475692 18722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:29:27.469870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.161.2:42599 (local address 127.27.161.1:37585)
0810 03:29:27.470102 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0810 03:29:27.470120 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0810 03:29:27.470485 (+ 365us) server_negotiation.cc:407] Connection header received
0810 03:29:27.470702 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:29:27.470707 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:29:27.470770 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:29:27.470844 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:29:27.471406 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:29:27.472121 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:29:27.474011 (+ 1890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:29:27.474783 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:29:27.474921 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:29:27.474998 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0810 03:29:27.475316 (+ 318us) server_negotiation.cc:299] Negotiation successful
0810 03:29:27.475440 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260810 03:29:27.475800 18735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:29:27.469491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.1:37585 (local address 127.27.161.2:42599)
0810 03:29:27.470364 (+ 873us) negotiation.cc:107] Waiting for socket to connect
0810 03:29:27.470384 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 03:29:27.470556 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:29:27.470882 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:29:27.470886 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:29:27.470913 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:29:27.471267 (+ 354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:29:27.471277 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:29:27.472262 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:29:27.472266 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:29:27.473833 (+ 1567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:29:27.473846 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:29:27.474999 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:29:27.475005 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:29:27.475142 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:29:27.475169 (+ 27us) client_negotiation.cc:769] Sending connection context
0810 03:29:27.475255 (+ 86us) client_negotiation.cc:240] Negotiation successful
0810 03:29:27.475393 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":744,"mutex_wait_us":47,"thread_start_us":523,"threads_started":1}
I20260810 03:29:27.475975 18718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:29:27.469555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.161.2:48951 (local address 127.27.161.3:46513)
0810 03:29:27.469781 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0810 03:29:27.469788 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:29:27.470725 (+ 937us) server_negotiation.cc:407] Connection header received
0810 03:29:27.470879 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:29:27.470883 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:29:27.470945 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:29:27.471017 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:29:27.471598 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:29:27.472441 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:29:27.474225 (+ 1784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:29:27.475010 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:29:27.475164 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:29:27.475235 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0810 03:29:27.475607 (+ 372us) server_negotiation.cc:299] Negotiation successful
0810 03:29:27.475733 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260810 03:29:27.476569 18736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:29:27.469605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.3:46513 (local address 127.27.161.2:48951)
0810 03:29:27.470607 (+ 1002us) negotiation.cc:107] Waiting for socket to connect
0810 03:29:27.470621 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 03:29:27.470748 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:29:27.471058 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:29:27.471062 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:29:27.471091 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:29:27.471447 (+ 356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:29:27.471457 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:29:27.472582 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:29:27.472585 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:29:27.474067 (+ 1482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:29:27.474080 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:29:27.475222 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:29:27.475229 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:29:27.475412 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:29:27.475445 (+ 33us) client_negotiation.cc:769] Sending connection context
0810 03:29:27.476152 (+ 707us) client_negotiation.cc:240] Negotiation successful
0810 03:29:27.476274 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":877,"thread_start_us":663,"threads_started":1}
I20260810 03:29:27.476897 18238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ccce1367a244b5f90bb820cfa222d9f" candidate_uuid: "6f9c1f7594aa43e5b1a4e3bd9dcdf7a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "868802ad74164cc8b84e264576a9bb05" is_pre_election: true
I20260810 03:29:27.477439 18238 raft_consensus.cc:2454] T 8ccce1367a244b5f90bb820cfa222d9f P 868802ad74164cc8b84e264576a9bb05 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6f9c1f7594aa43e5b1a4e3bd9dcdf7a1 in term 0.
I20260810 03:29:27.477474 18635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ccce1367a244b5f90bb820cfa222d9f" candidate_uuid: "6f9c1f7594aa43e5b1a4e3bd9dcdf7a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "386dbc17f7974e99a6a7500ccc1d16f8" is_pre_election: true
I20260810 03:29:27.477980 18635 raft_consensus.cc:2454] T 8ccce1367a244b5f90bb820cfa222d9f P 386dbc17f7974e99a6a7500ccc1d16f8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6f9c1f7594aa43e5b1a4e3bd9dcdf7a1 in term 0.
I20260810 03:29:27.478120 18339 leader_election.cc:302] T 8ccce1367a244b5f90bb820cfa222d9f P 6f9c1f7594aa43e5b1a4e3bd9dcdf7a1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6f9c1f7594aa43e5b1a4e3bd9dcdf7a1, 868802ad74164cc8b84e264576a9bb05; no voters:
I20260810 03:29:27.478526 18733 raft_consensus.cc:2781] T 8ccce1367a244b5f90bb820cfa222d9f P 6f9c1f7594aa43e5b1a4e3bd9dcdf7a1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 03:29:27.478655 18733 raft_consensus.cc:482] T 8ccce1367a244b5f90bb820cfa222d9f P 6f9c1f7594aa43e5b1a4e3bd9dcdf7a1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 03:29:27.478760 18733 raft_consensus.cc:3037] T 8ccce1367a244b5f90bb820cfa222d9f P 6f9c1f7594aa43e5b1a4e3bd9dcdf7a1 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:29:27.480917 18733 raft_consensus.cc:504] T 8ccce1367a244b5f90bb820cfa222d9f P 6f9c1f7594aa43e5b1a4e3bd9dcdf7a1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f9c1f7594aa43e5b1a4e3bd9dcdf7a1" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 45581 } } peers { permanent_uuid: "386dbc17f7974e99a6a7500ccc1d16f8" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 46513 } } peers { permanent_uuid: "868802ad74164cc8b84e264576a9bb05" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 37585 } }
I20260810 03:29:27.481518 18733 leader_election.cc:288] T 8ccce1367a244b5f90bb820cfa222d9f P 6f9c1f7594aa43e5b1a4e3bd9dcdf7a1 [CANDIDATE]: Term 1 election: Requested vote from peers 386dbc17f7974e99a6a7500ccc1d16f8 (127.27.161.3:46513), 868802ad74164cc8b84e264576a9bb05 (127.27.161.1:37585)
I20260810 03:29:27.482038 18635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ccce1367a244b5f90bb820cfa222d9f" candidate_uuid: "6f9c1f7594aa43e5b1a4e3bd9dcdf7a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "386dbc17f7974e99a6a7500ccc1d16f8"
I20260810 03:29:27.482087 18238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ccce1367a244b5f90bb820cfa222d9f" candidate_uuid: "6f9c1f7594aa43e5b1a4e3bd9dcdf7a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "868802ad74164cc8b84e264576a9bb05"
I20260810 03:29:27.482278 18238 raft_consensus.cc:3037] T 8ccce1367a244b5f90bb820cfa222d9f P 868802ad74164cc8b84e264576a9bb05 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:29:27.482278 18635 raft_consensus.cc:3037] T 8ccce1367a244b5f90bb820cfa222d9f P 386dbc17f7974e99a6a7500ccc1d16f8 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:29:27.485513 18238 raft_consensus.cc:2454] T 8ccce1367a244b5f90bb820cfa222d9f P 868802ad74164cc8b84e264576a9bb05 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6f9c1f7594aa43e5b1a4e3bd9dcdf7a1 in term 1.
I20260810 03:29:27.485513 18635 raft_consensus.cc:2454] T 8ccce1367a244b5f90bb820cfa222d9f P 386dbc17f7974e99a6a7500ccc1d16f8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6f9c1f7594aa43e5b1a4e3bd9dcdf7a1 in term 1.
I20260810 03:29:27.486006 18337 leader_election.cc:302] T 8ccce1367a244b5f90bb820cfa222d9f P 6f9c1f7594aa43e5b1a4e3bd9dcdf7a1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 386dbc17f7974e99a6a7500ccc1d16f8, 6f9c1f7594aa43e5b1a4e3bd9dcdf7a1, 868802ad74164cc8b84e264576a9bb05; no voters:
I20260810 03:29:27.486393 18733 raft_consensus.cc:2781] T 8ccce1367a244b5f90bb820cfa222d9f P 6f9c1f7594aa43e5b1a4e3bd9dcdf7a1 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:29:27.486825 18733 raft_consensus.cc:686] T 8ccce1367a244b5f90bb820cfa222d9f P 6f9c1f7594aa43e5b1a4e3bd9dcdf7a1 [term 1 LEADER]: Becoming Leader. State: Replica: 6f9c1f7594aa43e5b1a4e3bd9dcdf7a1, State: Running, Role: LEADER
I20260810 03:29:27.487332 18733 consensus_queue.cc:237] T 8ccce1367a244b5f90bb820cfa222d9f P 6f9c1f7594aa43e5b1a4e3bd9dcdf7a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f9c1f7594aa43e5b1a4e3bd9dcdf7a1" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 45581 } } peers { permanent_uuid: "386dbc17f7974e99a6a7500ccc1d16f8" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 46513 } } peers { permanent_uuid: "868802ad74164cc8b84e264576a9bb05" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 37585 } }
I20260810 03:29:27.491802 18009 catalog_manager.cc:5495] T 8ccce1367a244b5f90bb820cfa222d9f P 6f9c1f7594aa43e5b1a4e3bd9dcdf7a1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6f9c1f7594aa43e5b1a4e3bd9dcdf7a1 (127.27.161.2). New cstate: current_term: 1 leader_uuid: "6f9c1f7594aa43e5b1a4e3bd9dcdf7a1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f9c1f7594aa43e5b1a4e3bd9dcdf7a1" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 45581 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "386dbc17f7974e99a6a7500ccc1d16f8" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 46513 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "868802ad74164cc8b84e264576a9bb05" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 37585 } health_report { overall_health: UNKNOWN } } }
W20260810 03:29:27.504201 18306 tablet.cc:2369] T 8ccce1367a244b5f90bb820cfa222d9f P 868802ad74164cc8b84e264576a9bb05: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 03:29:27.531167 18715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:29:27.525741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39200 (local address 127.27.161.2:45581)
0810 03:29:27.525998 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0810 03:29:27.526004 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:29:27.526024 (+ 20us) server_negotiation.cc:407] Connection header received
0810 03:29:27.526228 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:29:27.526232 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:29:27.526313 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:29:27.526448 (+ 135us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 03:29:27.526822 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:29:27.527543 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:29:27.528929 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:29:27.529350 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:29:27.529461 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:29:27.530014 (+ 553us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 03:29:27.530566 (+ 552us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 03:29:27.530713 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0810 03:29:27.530817 (+ 104us) server_negotiation.cc:299] Negotiation successful
0810 03:29:27.530902 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260810 03:29:27.539354 18635 raft_consensus.cc:1261] T 8ccce1367a244b5f90bb820cfa222d9f P 386dbc17f7974e99a6a7500ccc1d16f8 [term 1 FOLLOWER]: Refusing update from remote peer 6f9c1f7594aa43e5b1a4e3bd9dcdf7a1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:29:27.539358 18238 raft_consensus.cc:1261] T 8ccce1367a244b5f90bb820cfa222d9f P 868802ad74164cc8b84e264576a9bb05 [term 1 FOLLOWER]: Refusing update from remote peer 6f9c1f7594aa43e5b1a4e3bd9dcdf7a1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:29:27.540329 18733 consensus_queue.cc:1035] T 8ccce1367a244b5f90bb820cfa222d9f P 6f9c1f7594aa43e5b1a4e3bd9dcdf7a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "386dbc17f7974e99a6a7500ccc1d16f8" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 46513 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:29:27.540673 18738 consensus_queue.cc:1035] T 8ccce1367a244b5f90bb820cfa222d9f P 6f9c1f7594aa43e5b1a4e3bd9dcdf7a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "868802ad74164cc8b84e264576a9bb05" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 37585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:29:27.552971 18742 mvcc.cc:205] Tried to move back new op lower bound from 7316818196630982656 to 7316818196431888384. Current Snapshot: MvccSnapshot[applied={T|T < 7316818196630982656}]
I20260810 03:29:27.564556 18008 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:58250:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 03:29:27.564817 18008 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:58250:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 03:29:27.568485 18008 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 104edb053ce54ee1881fb2cbcace28cb: Sending DeleteTablet for 3 replicas of tablet 8ccce1367a244b5f90bb820cfa222d9f
I20260810 03:29:27.569651 18609 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ccce1367a244b5f90bb820cfa222d9f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:29:27 UTC) from {username='kudu', principal='kudu/127.27.161.62@KRBTEST.COM'} at 127.0.0.1:36398
I20260810 03:29:27.569646 18401 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ccce1367a244b5f90bb820cfa222d9f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:29:27 UTC) from {username='kudu', principal='kudu/127.27.161.62@KRBTEST.COM'} at 127.0.0.1:39190
I20260810 03:29:27.570420 18198 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ccce1367a244b5f90bb820cfa222d9f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:29:27 UTC) from {username='kudu', principal='kudu/127.27.161.62@KRBTEST.COM'} at 127.0.0.1:36956
I20260810 03:29:27.570544 18758 tablet_replica.cc:330] T 8ccce1367a244b5f90bb820cfa222d9f P 386dbc17f7974e99a6a7500ccc1d16f8: stopping tablet replica
I20260810 03:29:27.570990 18758 raft_consensus.cc:2229] T 8ccce1367a244b5f90bb820cfa222d9f P 386dbc17f7974e99a6a7500ccc1d16f8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 03:29:27.571525 18758 raft_consensus.cc:2258] T 8ccce1367a244b5f90bb820cfa222d9f P 386dbc17f7974e99a6a7500ccc1d16f8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:29:27.571780 28292 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iBH5/build/asan/bin/kudu pid 18066
I20260810 03:29:27.572876 18760 tablet_replica.cc:330] T 8ccce1367a244b5f90bb820cfa222d9f P 868802ad74164cc8b84e264576a9bb05: stopping tablet replica
I20260810 03:29:27.573331 18760 raft_consensus.cc:2229] T 8ccce1367a244b5f90bb820cfa222d9f P 868802ad74164cc8b84e264576a9bb05 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 03:29:27.573908 18760 raft_consensus.cc:2258] T 8ccce1367a244b5f90bb820cfa222d9f P 868802ad74164cc8b84e264576a9bb05 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:29:27.574247 18758 ts_tablet_manager.cc:1905] T 8ccce1367a244b5f90bb820cfa222d9f P 386dbc17f7974e99a6a7500ccc1d16f8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:29:27.574441 18759 tablet_replica.cc:330] T 8ccce1367a244b5f90bb820cfa222d9f P 6f9c1f7594aa43e5b1a4e3bd9dcdf7a1: stopping tablet replica
I20260810 03:29:27.574887 18759 raft_consensus.cc:2229] T 8ccce1367a244b5f90bb820cfa222d9f P 6f9c1f7594aa43e5b1a4e3bd9dcdf7a1 [term 1 LEADER]: Raft consensus shutting down.
I20260810 03:29:27.575567 18759 raft_consensus.cc:2258] T 8ccce1367a244b5f90bb820cfa222d9f P 6f9c1f7594aa43e5b1a4e3bd9dcdf7a1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:29:27.580200 18760 ts_tablet_manager.cc:1905] T 8ccce1367a244b5f90bb820cfa222d9f P 868802ad74164cc8b84e264576a9bb05: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:29:27.581367 18758 ts_tablet_manager.cc:1918] T 8ccce1367a244b5f90bb820cfa222d9f P 386dbc17f7974e99a6a7500ccc1d16f8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 03:29:27.581621 18758 log.cc:1196] T 8ccce1367a244b5f90bb820cfa222d9f P 386dbc17f7974e99a6a7500ccc1d16f8: Deleting WAL directory at /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/minicluster-data/ts-2/wal/wals/8ccce1367a244b5f90bb820cfa222d9f
I20260810 03:29:27.582341 18758 ts_tablet_manager.cc:1939] T 8ccce1367a244b5f90bb820cfa222d9f P 386dbc17f7974e99a6a7500ccc1d16f8: Deleting consensus metadata
I20260810 03:29:27.583321 18759 ts_tablet_manager.cc:1905] T 8ccce1367a244b5f90bb820cfa222d9f P 6f9c1f7594aa43e5b1a4e3bd9dcdf7a1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:29:27.584596 17991 catalog_manager.cc:4841] TS 386dbc17f7974e99a6a7500ccc1d16f8 (127.27.161.3:46513): tablet 8ccce1367a244b5f90bb820cfa222d9f (table test-table [id=feffebde7395446b9d00f5a22ddd2b85]) successfully deleted
I20260810 03:29:27.589241 18760 ts_tablet_manager.cc:1918] T 8ccce1367a244b5f90bb820cfa222d9f P 868802ad74164cc8b84e264576a9bb05: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 03:29:27.589525 18760 log.cc:1196] T 8ccce1367a244b5f90bb820cfa222d9f P 868802ad74164cc8b84e264576a9bb05: Deleting WAL directory at /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/minicluster-data/ts-0/wal/wals/8ccce1367a244b5f90bb820cfa222d9f
I20260810 03:29:27.590232 18760 ts_tablet_manager.cc:1939] T 8ccce1367a244b5f90bb820cfa222d9f P 868802ad74164cc8b84e264576a9bb05: Deleting consensus metadata
I20260810 03:29:27.591112 18722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:29:27.573227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36968 (local address 127.27.161.1:37585)
0810 03:29:27.573397 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0810 03:29:27.573403 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:29:27.573421 (+ 18us) server_negotiation.cc:407] Connection header received
0810 03:29:27.573485 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:29:27.573489 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:29:27.573570 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:29:27.573674 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:29:27.575993 (+ 2319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:29:27.577122 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:29:27.578968 (+ 1846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:29:27.579588 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:29:27.579739 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:29:27.583285 (+ 3546us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:29:27.583303 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:29:27.583322 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:29:27.583385 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:29:27.587455 (+ 4070us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:29:27.588475 (+ 1020us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:29:27.588485 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:29:27.588488 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:29:27.588675 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:29:27.589116 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:29:27.589122 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:29:27.589126 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:29:27.589710 (+ 584us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:29:27.590063 (+ 353us) server_negotiation.cc:1035] Waiting for connection context
0810 03:29:27.590444 (+ 381us) server_negotiation.cc:299] Negotiation successful
0810 03:29:27.590717 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260810 03:29:27.592078 18298 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:29:27.592273 17993 catalog_manager.cc:4841] TS 868802ad74164cc8b84e264576a9bb05 (127.27.161.1:37585): tablet 8ccce1367a244b5f90bb820cfa222d9f (table test-table [id=feffebde7395446b9d00f5a22ddd2b85]) successfully deleted
I20260810 03:29:27.594907 18759 ts_tablet_manager.cc:1918] T 8ccce1367a244b5f90bb820cfa222d9f P 6f9c1f7594aa43e5b1a4e3bd9dcdf7a1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 03:29:27.595193 18759 log.cc:1196] T 8ccce1367a244b5f90bb820cfa222d9f P 6f9c1f7594aa43e5b1a4e3bd9dcdf7a1: Deleting WAL directory at /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/minicluster-data/ts-1/wal/wals/8ccce1367a244b5f90bb820cfa222d9f
I20260810 03:29:27.596040 18759 ts_tablet_manager.cc:1939] T 8ccce1367a244b5f90bb820cfa222d9f P 6f9c1f7594aa43e5b1a4e3bd9dcdf7a1: Deleting consensus metadata
I20260810 03:29:27.598140 17986 catalog_manager.cc:4841] TS 6f9c1f7594aa43e5b1a4e3bd9dcdf7a1 (127.27.161.2:45581): tablet 8ccce1367a244b5f90bb820cfa222d9f (table test-table [id=feffebde7395446b9d00f5a22ddd2b85]) successfully deleted
I20260810 03:29:27.682636 28292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iBH5/build/asan/bin/kudu with pid 18066
I20260810 03:29:27.708606 28292 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iBH5/build/asan/bin/kudu pid 18310
I20260810 03:29:27.727955 18715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:29:27.709776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39214 (local address 127.27.161.2:45581)
0810 03:29:27.710076 (+ 300us) server_negotiation.cc:206] Beginning negotiation
0810 03:29:27.710083 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:29:27.710108 (+ 25us) server_negotiation.cc:407] Connection header received
0810 03:29:27.710172 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:29:27.710177 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:29:27.710262 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:29:27.710364 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:29:27.712072 (+ 1708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:29:27.713496 (+ 1424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:29:27.715624 (+ 2128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:29:27.716357 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:29:27.716503 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:29:27.719999 (+ 3496us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:29:27.720015 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:29:27.720035 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:29:27.720101 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:29:27.724423 (+ 4322us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:29:27.725336 (+ 913us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:29:27.725346 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:29:27.725350 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:29:27.725562 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:29:27.726035 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:29:27.726040 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:29:27.726044 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:29:27.726654 (+ 610us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:29:27.727017 (+ 363us) server_negotiation.cc:1035] Waiting for connection context
0810 03:29:27.727307 (+ 290us) server_negotiation.cc:299] Negotiation successful
0810 03:29:27.727553 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260810 03:29:27.728794 18487 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:29:27.842826 28292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iBH5/build/asan/bin/kudu with pid 18310
I20260810 03:29:27.869330 28292 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iBH5/build/asan/bin/kudu pid 18500
I20260810 03:29:27.882792 18718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:29:27.870160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36414 (local address 127.27.161.3:46513)
0810 03:29:27.870349 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0810 03:29:27.870353 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:29:27.870373 (+ 20us) server_negotiation.cc:407] Connection header received
0810 03:29:27.870508 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:29:27.870512 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:29:27.870580 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:29:27.870702 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:29:27.871985 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:29:27.872698 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:29:27.874108 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:29:27.874504 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:29:27.874603 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:29:27.877255 (+ 2652us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:29:27.877272 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:29:27.877291 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:29:27.877326 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:29:27.879984 (+ 2658us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:29:27.880715 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:29:27.880723 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:29:27.880726 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:29:27.880875 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:29:27.881169 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:29:27.881172 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:29:27.881173 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:29:27.881666 (+ 493us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:29:27.881902 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0810 03:29:27.882304 (+ 402us) server_negotiation.cc:299] Negotiation successful
0810 03:29:27.882483 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260810 03:29:27.883507 18691 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:29:27.972537 28292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iBH5/build/asan/bin/kudu with pid 18500
I20260810 03:29:27.998394 28292 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iBH5/build/asan/bin/kudu pid 17964
I20260810 03:29:27.999625 18055 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:29:28.116993 28292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iBH5/build/asan/bin/kudu with pid 17964
I20260810 03:29:28.146142 28292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iBH5/build/asan/bin/kudu
/tmp/dist-test-task24iBH5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.161.62:35141
--webserver_interface=127.27.161.62
--webserver_port=35053
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.62.keytab
--principal=kudu/127.27.161.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.161.20:36555
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.161.62:35141
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:29:28.509583 18765 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:29:28.509900 18765 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:29:28.510052 18765 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:29:28.517541 18765 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:29:28.517671 18765 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:29:28.517735 18765 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:29:28.517817 18765 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:29:28.517874 18765 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:29:28.528328 18765 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.161.20:36555
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.161.62:35141
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.62.keytab
--principal=kudu/127.27.161.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.161.62:35141
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.161.62
--webserver_port=35053
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:29:28.529742 18765 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:29:28.531090 18765 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:29:28.542981 18770 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:29:28.542981 18771 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:29:28.543058 18774 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:29:28.543992 18765 server_base.cc:1034] running on GCE node
I20260810 03:29:28.544768 18765 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:29:28.546248 18765 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:29:28.547484 18765 hybrid_clock.cc:648] HybridClock initialized: now 1786332568547413 us; error 51 us; skew 500 ppm
I20260810 03:29:28.547848 18765 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:29:28 dist-test-slave-vpvm krb5kdc[17099](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332568, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.161.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:29:28.550962 18765 init.cc:377] Logged in from keytab as kudu/127.27.161.62@KRBTEST.COM (short username kudu)
I20260810 03:29:28.552500 18765 webserver.cc:460] Webserver started at http://127.27.161.62:35053/ using document root <none> and password file <none>
I20260810 03:29:28.553205 18765 fs_manager.cc:359] Metadata directory not provided
I20260810 03:29:28.553314 18765 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:29:28.559350 18765 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260810 03:29:28.562546 18784 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:29:28.564083 18765 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260810 03:29:28.564291 18765 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/minicluster-data/master-0/data,/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/minicluster-data/master-0/wal
uuid: "104edb053ce54ee1881fb2cbcace28cb"
format_stamp: "Formatted at 2026-08-10 03:29:22 on dist-test-slave-vpvm"
server_key: "aff4ca34f5f297d99297f8303e9c1a5b"
server_key_iv: "eab5ab9508a75442c91afc3bc3405470"
server_key_version: "encryptionkey@0"
I20260810 03:29:28.565317 18765 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:29:28.593511 18765 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:29:28.597895 18765 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:29:28.598284 18765 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:29:28.616956 18765 rpc_server.cc:307] RPC server started. Bound to: 127.27.161.62:35141
I20260810 03:29:28.616964 18877 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.161.62:35141 every 8 connection(s)
I20260810 03:29:28.618525 18765 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/minicluster-data/master-0/data/info.pb
I20260810 03:29:28.625759 18878 sys_catalog.cc:263] Verifying existing consensus state
I20260810 03:29:28.628417 28292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iBH5/build/asan/bin/kudu as pid 18765
I20260810 03:29:28.629295 28292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iBH5/build/asan/bin/kudu
/tmp/dist-test-task24iBH5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.161.1:37585
--local_ip_for_outbound_sockets=127.27.161.1
--tserver_master_addrs=127.27.161.62:35141
--webserver_port=37695
--webserver_interface=127.27.161.1
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.1.keytab
--principal=kudu/127.27.161.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.161.20:36555
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 03:29:28.629616 18878 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 104edb053ce54ee1881fb2cbcace28cb: Bootstrap starting.
I20260810 03:29:28.652987 18878 log.cc:824] T 00000000000000000000000000000000 P 104edb053ce54ee1881fb2cbcace28cb: Log is configured to *not* fsync() on all Append() calls
I20260810 03:29:28.677009 18878 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 104edb053ce54ee1881fb2cbcace28cb: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 03:29:28.677769 18878 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 104edb053ce54ee1881fb2cbcace28cb: Bootstrap complete.
I20260810 03:29:28.691483 18878 raft_consensus.cc:348] T 00000000000000000000000000000000 P 104edb053ce54ee1881fb2cbcace28cb [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "104edb053ce54ee1881fb2cbcace28cb" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 35141 } }
I20260810 03:29:28.692746 18878 raft_consensus.cc:729] T 00000000000000000000000000000000 P 104edb053ce54ee1881fb2cbcace28cb [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 104edb053ce54ee1881fb2cbcace28cb, State: Initialized, Role: FOLLOWER
I20260810 03:29:28.693506 18878 consensus_queue.cc:260] T 00000000000000000000000000000000 P 104edb053ce54ee1881fb2cbcace28cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "104edb053ce54ee1881fb2cbcace28cb" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 35141 } }
I20260810 03:29:28.693804 18878 raft_consensus.cc:388] T 00000000000000000000000000000000 P 104edb053ce54ee1881fb2cbcace28cb [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 03:29:28.693933 18878 raft_consensus.cc:482] T 00000000000000000000000000000000 P 104edb053ce54ee1881fb2cbcace28cb [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 03:29:28.694149 18878 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 104edb053ce54ee1881fb2cbcace28cb [term 2 FOLLOWER]: Advancing to term 3
I20260810 03:29:28.700081 18878 raft_consensus.cc:504] T 00000000000000000000000000000000 P 104edb053ce54ee1881fb2cbcace28cb [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "104edb053ce54ee1881fb2cbcace28cb" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 35141 } }
I20260810 03:29:28.700742 18878 leader_election.cc:302] T 00000000000000000000000000000000 P 104edb053ce54ee1881fb2cbcace28cb [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 104edb053ce54ee1881fb2cbcace28cb; no voters:
I20260810 03:29:28.701465 18878 leader_election.cc:288] T 00000000000000000000000000000000 P 104edb053ce54ee1881fb2cbcace28cb [CANDIDATE]: Term 3 election: Requested vote from peers
I20260810 03:29:28.701634 18883 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 104edb053ce54ee1881fb2cbcace28cb [term 3 FOLLOWER]: Leader election won for term 3
I20260810 03:29:28.702778 18883 raft_consensus.cc:686] T 00000000000000000000000000000000 P 104edb053ce54ee1881fb2cbcace28cb [term 3 LEADER]: Becoming Leader. State: Replica: 104edb053ce54ee1881fb2cbcace28cb, State: Running, Role: LEADER
I20260810 03:29:28.703374 18883 consensus_queue.cc:237] T 00000000000000000000000000000000 P 104edb053ce54ee1881fb2cbcace28cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "104edb053ce54ee1881fb2cbcace28cb" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 35141 } }
I20260810 03:29:28.703738 18878 sys_catalog.cc:564] T 00000000000000000000000000000000 P 104edb053ce54ee1881fb2cbcace28cb [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:29:28.705874 18884 sys_catalog.cc:455] T 00000000000000000000000000000000 P 104edb053ce54ee1881fb2cbcace28cb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "104edb053ce54ee1881fb2cbcace28cb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "104edb053ce54ee1881fb2cbcace28cb" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 35141 } } }
I20260810 03:29:28.705852 18885 sys_catalog.cc:455] T 00000000000000000000000000000000 P 104edb053ce54ee1881fb2cbcace28cb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 104edb053ce54ee1881fb2cbcace28cb. Latest consensus state: current_term: 3 leader_uuid: "104edb053ce54ee1881fb2cbcace28cb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "104edb053ce54ee1881fb2cbcace28cb" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 35141 } } }
I20260810 03:29:28.706359 18885 sys_catalog.cc:458] T 00000000000000000000000000000000 P 104edb053ce54ee1881fb2cbcace28cb [sys.catalog]: This master's current role is: LEADER
I20260810 03:29:28.706362 18884 sys_catalog.cc:458] T 00000000000000000000000000000000 P 104edb053ce54ee1881fb2cbcace28cb [sys.catalog]: This master's current role is: LEADER
I20260810 03:29:28.707329 18891 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 03:29:28.726261 18891 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 03:29:28.727964 18891 catalog_manager.cc:1259] Loaded cluster ID: 04b129a66593464187ede0ccb2dd4272
I20260810 03:29:28.728175 18891 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260810 03:29:28.730587 18891 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7ff2bef789ad google::LogMessage::Fail() at ??:0
@ 0x7ff2bef7b88a google::LogMessage::SendToLog() at ??:0
@ 0x7ff2bef784de google::LogMessage::Flush() at ??:0
@ 0x7ff2bef79a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7ff2d49ab512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7ff2d49a8976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7ff2d4a30610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7ff2d4a30439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7ff2c1bc4345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7ff2c1bc791c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7ff2c1bc7799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7ff2c1ba2eaf kudu::Thread::SuperviseThread() at ??:0
I20260810 03:29:28.792531 18916 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
@ 0x7ff2c4e7c6db start_thread at ??:0
@ 0x7ff2bde1261f clone at ??:0
W20260810 03:29:29.142673 18881 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:29:29.142994 18881 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:29:29.143102 18881 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:29:29.150624 18881 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:29:29.150753 18881 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:29:29.150869 18881 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.161.1
I20260810 03:29:29.161154 18881 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.161.20:36555
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.1.keytab
--principal=kudu/127.27.161.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.161.1:37585
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.161.1
--webserver_port=37695
--tserver_master_addrs=127.27.161.62:35141
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.161.1
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:29:29.162559 18881 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:29:29.163966 18881 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:29:29.177763 18938 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:29:29.177734 18940 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:29:29.177722 18945 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:29:29.178678 18881 server_base.cc:1034] running on GCE node
I20260810 03:29:29.179345 18881 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:29:29.180378 18881 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:29:29.181607 18881 hybrid_clock.cc:648] HybridClock initialized: now 1786332569181540 us; error 44 us; skew 500 ppm
I20260810 03:29:29.181982 18881 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:29:29 dist-test-slave-vpvm krb5kdc[17099](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332569, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.161.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:29:29.185379 18881 init.cc:377] Logged in from keytab as kudu/127.27.161.1@KRBTEST.COM (short username kudu)
I20260810 03:29:29.187105 18881 webserver.cc:460] Webserver started at http://127.27.161.1:37695/ using document root <none> and password file <none>
I20260810 03:29:29.187920 18881 fs_manager.cc:359] Metadata directory not provided
I20260810 03:29:29.188048 18881 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:29:29.194000 18881 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260810 03:29:29.196988 18955 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:29:29.198755 18881 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260810 03:29:29.198946 18881 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/minicluster-data/ts-0/data,/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/minicluster-data/ts-0/wal
uuid: "868802ad74164cc8b84e264576a9bb05"
format_stamp: "Formatted at 2026-08-10 03:29:23 on dist-test-slave-vpvm"
server_key: "844fdd946a70ccfd1a823157e16e41cb"
server_key_iv: "c4bddeddb2a89fbf0690af36a364c10e"
server_key_version: "encryptionkey@0"
I20260810 03:29:29.200240 18881 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:29:29.226982 18881 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:29:29.231695 18881 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:29:29.231995 18881 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:29:29.233143 18881 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:29:29.234869 18881 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:29:29.234992 18881 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:29:29.235116 18881 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:29:29.235203 18881 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:29:29.269902 18881 rpc_server.cc:307] RPC server started. Bound to: 127.27.161.1:37585
I20260810 03:29:29.270013 19157 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.161.1:37585 every 8 connection(s)
I20260810 03:29:29.271474 18881 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/minicluster-data/ts-0/data/info.pb
I20260810 03:29:29.276461 19164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:29:29.274595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:35141 (local address 127.27.161.1:58793)
0810 03:29:29.275354 (+ 759us) negotiation.cc:107] Waiting for socket to connect
0810 03:29:29.275591 (+ 237us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.62:35141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":508,"thread_start_us":292,"threads_started":1}
W20260810 03:29:29.277122 19160 heartbeater.cc:644] Failed to heartbeat to 127.27.161.62:35141 (0 consecutive failures): Network error: Failed to ping master at 127.27.161.62:35141: Client connection negotiation failed: client connection to 127.27.161.62:35141: connect: Connection refused (error 111)
I20260810 03:29:29.277917 19164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:29:29.277608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:35141 (local address 127.27.161.1:41073)
0810 03:29:29.277739 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0810 03:29:29.277808 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.62:35141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260810 03:29:29.278687 19164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:29:29.278399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:35141 (local address 127.27.161.1:52337)
0810 03:29:29.278507 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0810 03:29:29.278578 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.62:35141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260810 03:29:29.278887 19160 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260810 03:29:29.279675 28292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iBH5/build/asan/bin/kudu as pid 18881
I20260810 03:29:29.280258 28292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iBH5/build/asan/bin/kudu
/tmp/dist-test-task24iBH5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.161.2:45581
--local_ip_for_outbound_sockets=127.27.161.2
--tserver_master_addrs=127.27.161.62:35141
--webserver_port=34879
--webserver_interface=127.27.161.2
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.2.keytab
--principal=kudu/127.27.161.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.161.20:36555
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:29:29.637995 19166 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:29:29.638336 19166 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:29:29.638438 19166 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:29:29.645937 19166 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:29:29.646104 19166 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:29:29.646240 19166 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.161.2
I20260810 03:29:29.656430 19166 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.161.20:36555
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.2.keytab
--principal=kudu/127.27.161.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.161.2:45581
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.161.2
--webserver_port=34879
--tserver_master_addrs=127.27.161.62:35141
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.161.2
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:29:29.657805 19166 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:29:29.659320 19166 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:29:29.673207 19175 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:29:29.673120 19180 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:29:29.673202 19177 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:29:29.674266 19166 server_base.cc:1034] running on GCE node
I20260810 03:29:29.674938 19166 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:29:29.675947 19166 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:29:29.677177 19166 hybrid_clock.cc:648] HybridClock initialized: now 1786332569677124 us; error 43 us; skew 500 ppm
I20260810 03:29:29.677541 19166 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:29:29 dist-test-slave-vpvm krb5kdc[17099](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332569, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.161.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:29:29.681025 19166 init.cc:377] Logged in from keytab as kudu/127.27.161.2@KRBTEST.COM (short username kudu)
I20260810 03:29:29.682682 19166 webserver.cc:460] Webserver started at http://127.27.161.2:34879/ using document root <none> and password file <none>
I20260810 03:29:29.683441 19166 fs_manager.cc:359] Metadata directory not provided
I20260810 03:29:29.683568 19166 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:29:29.689311 19166 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260810 03:29:29.692386 19198 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:29:29.693821 19166 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 03:29:29.694032 19166 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/minicluster-data/ts-1/data,/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/minicluster-data/ts-1/wal
uuid: "6f9c1f7594aa43e5b1a4e3bd9dcdf7a1"
format_stamp: "Formatted at 2026-08-10 03:29:23 on dist-test-slave-vpvm"
server_key: "8ae896ece668e6bb6925e22796815542"
server_key_iv: "ff012c096deb7a32414389673d403323"
server_key_version: "encryptionkey@0"
I20260810 03:29:29.695094 19166 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:29:29.717012 19166 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:29:29.722021 19166 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:29:29.722316 19166 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:29:29.723436 19166 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:29:29.724920 19166 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:29:29.725035 19166 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:29:29.725154 19166 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:29:29.725222 19166 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:29:29.759501 19166 rpc_server.cc:307] RPC server started. Bound to: 127.27.161.2:45581
I20260810 03:29:29.759568 19372 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.161.2:45581 every 8 connection(s)
I20260810 03:29:29.761304 19166 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/minicluster-data/ts-1/data/info.pb
I20260810 03:29:29.762359 28292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iBH5/build/asan/bin/kudu as pid 19166
I20260810 03:29:29.763145 28292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iBH5/build/asan/bin/kudu
/tmp/dist-test-task24iBH5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.161.3:46513
--local_ip_for_outbound_sockets=127.27.161.3
--tserver_master_addrs=127.27.161.62:35141
--webserver_port=33479
--webserver_interface=127.27.161.3
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.3.keytab
--principal=kudu/127.27.161.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.161.20:36555
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 03:29:29.767359 19375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:29:29.764927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:35141 (local address 127.27.161.2:53535)
0810 03:29:29.765818 (+ 891us) negotiation.cc:107] Waiting for socket to connect
0810 03:29:29.766167 (+ 349us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.62:35141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":565,"thread_start_us":333,"threads_started":1}
W20260810 03:29:29.768193 19373 heartbeater.cc:644] Failed to heartbeat to 127.27.161.62:35141 (0 consecutive failures): Network error: Failed to ping master at 127.27.161.62:35141: Client connection negotiation failed: client connection to 127.27.161.62:35141: connect: Connection refused (error 111)
I20260810 03:29:29.769238 19375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:29:29.768818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:35141 (local address 127.27.161.2:39955)
0810 03:29:29.768997 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0810 03:29:29.769092 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.62:35141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260810 03:29:29.770287 19375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:29:29.769871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:35141 (local address 127.27.161.2:47195)
0810 03:29:29.770046 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0810 03:29:29.770147 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.62:35141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260810 03:29:29.770579 19373 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260810 03:29:30.129009 19376 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:29:30.129333 19376 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:29:30.129469 19376 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:29:30.137010 19376 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:29:30.137146 19376 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:29:30.137300 19376 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.161.3
I20260810 03:29:30.147598 19376 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.161.20:36555
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.3.keytab
--principal=kudu/127.27.161.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.161.3:46513
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.161.3
--webserver_port=33479
--tserver_master_addrs=127.27.161.62:35141
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.161.3
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:29:30.148972 19376 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:29:30.150386 19376 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:29:30.163862 19385 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:29:30.163931 19386 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:29:30.163959 19390 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:29:30.164788 19376 server_base.cc:1034] running on GCE node
I20260810 03:29:30.165501 19376 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:29:30.166580 19376 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:29:30.167794 19376 hybrid_clock.cc:648] HybridClock initialized: now 1786332570167744 us; error 44 us; skew 500 ppm
I20260810 03:29:30.168129 19376 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:29:30 dist-test-slave-vpvm krb5kdc[17099](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332570, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.161.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:29:30.171446 19376 init.cc:377] Logged in from keytab as kudu/127.27.161.3@KRBTEST.COM (short username kudu)
I20260810 03:29:30.172946 19376 webserver.cc:460] Webserver started at http://127.27.161.3:33479/ using document root <none> and password file <none>
I20260810 03:29:30.173669 19376 fs_manager.cc:359] Metadata directory not provided
I20260810 03:29:30.173795 19376 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:29:30.179576 19376 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.000s sys 0.004s
I20260810 03:29:30.182741 19402 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:29:30.184310 19376 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 03:29:30.184507 19376 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/minicluster-data/ts-2/data,/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/minicluster-data/ts-2/wal
uuid: "386dbc17f7974e99a6a7500ccc1d16f8"
format_stamp: "Formatted at 2026-08-10 03:29:24 on dist-test-slave-vpvm"
server_key: "c10c20a349ff82746b8dc4eca2a2671b"
server_key_iv: "f0e46891c61ae3b300c0791780f9daeb"
server_key_version: "encryptionkey@0"
I20260810 03:29:30.185590 19376 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:29:30.203459 19376 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:29:30.208581 19376 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:29:30.208871 19376 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:29:30.210024 19376 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:29:30.211547 19376 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:29:30.211658 19376 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:29:30.211771 19376 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:29:30.211846 19376 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:29:30.245141 19376 rpc_server.cc:307] RPC server started. Bound to: 127.27.161.3:46513
I20260810 03:29:30.245203 19573 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.161.3:46513 every 8 connection(s)
I20260810 03:29:30.246779 19376 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786332336319544-28292-0/minicluster-data/ts-2/data/info.pb
I20260810 03:29:30.252902 19579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:29:30.250383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:35141 (local address 127.27.161.3:38107)
0810 03:29:30.251329 (+ 946us) negotiation.cc:107] Waiting for socket to connect
0810 03:29:30.251637 (+ 308us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.62:35141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":598,"thread_start_us":348,"threads_started":1}
W20260810 03:29:30.253602 19575 heartbeater.cc:644] Failed to heartbeat to 127.27.161.62:35141 (0 consecutive failures): Network error: Failed to ping master at 127.27.161.62:35141: Client connection negotiation failed: client connection to 127.27.161.62:35141: connect: Connection refused (error 111)
I20260810 03:29:30.255146 19579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:29:30.254753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:35141 (local address 127.27.161.3:42473)
0810 03:29:30.254923 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0810 03:29:30.255015 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.62:35141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260810 03:29:30.256173 19579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:29:30.255771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:35141 (local address 127.27.161.3:54811)
0810 03:29:30.255939 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0810 03:29:30.256029 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.62:35141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260810 03:29:30.256469 19575 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260810 03:29:30.256875 28292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iBH5/build/asan/bin/kudu as pid 19376
I20260810 03:29:30.280505 19584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:29:30.279609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:35141 (local address 127.27.161.1:34195)
0810 03:29:30.280129 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0810 03:29:30.280255 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.62:35141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":247,"threads_started":1}
I20260810 03:29:30.772351 19585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:29:30.771335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:35141 (local address 127.27.161.2:53831)
0810 03:29:30.771931 (+ 596us) negotiation.cc:107] Waiting for socket to connect
0810 03:29:30.772068 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.62:35141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":285,"threads_started":1}
I20260810 03:29:31.258037 19588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:29:31.257130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:35141 (local address 127.27.161.3:51641)
0810 03:29:31.257624 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0810 03:29:31.257756 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.62:35141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":236,"threads_started":1}
I20260810 03:29:31.282476 19589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:29:31.281613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:35141 (local address 127.27.161.1:43123)
0810 03:29:31.282214 (+ 601us) negotiation.cc:107] Waiting for socket to connect
0810 03:29:31.282304 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.62:35141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":259,"threads_started":1}
I20260810 03:29:31.774420 19590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:29:31.773577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:35141 (local address 127.27.161.2:53779)
0810 03:29:31.774143 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0810 03:29:31.774241 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.62:35141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":259,"threads_started":1}
I20260810 03:29:32.260026 19591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:29:32.259143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:35141 (local address 127.27.161.3:43575)
0810 03:29:32.259747 (+ 604us) negotiation.cc:107] Waiting for socket to connect
0810 03:29:32.259845 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.62:35141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":293,"threads_started":1}
I20260810 03:29:32.284427 19592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:29:32.283581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:35141 (local address 127.27.161.1:53365)
0810 03:29:32.284118 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0810 03:29:32.284220 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.62:35141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":265,"threads_started":1}
I20260810 03:29:32.776289 19595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:29:32.775480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:35141 (local address 127.27.161.2:48629)
0810 03:29:32.775968 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0810 03:29:32.776062 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.62:35141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":249,"threads_started":1}
I20260810 03:29:33.261857 19596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:29:33.261054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:35141 (local address 127.27.161.3:44007)
0810 03:29:33.261577 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0810 03:29:33.261675 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.62:35141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":260,"threads_started":1}
I20260810 03:29:33.286244 19601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:29:33.285442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:35141 (local address 127.27.161.1:36729)
0810 03:29:33.285945 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0810 03:29:33.286066 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.62:35141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":236,"threads_started":1}
I20260810 03:29:33.778136 19602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:29:33.777282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:35141 (local address 127.27.161.2:40171)
0810 03:29:33.777801 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0810 03:29:33.777897 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.62:35141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":267,"threads_started":1}
I20260810 03:29:34.263628 19603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:29:34.262858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:35141 (local address 127.27.161.3:39773)
0810 03:29:34.263314 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0810 03:29:34.263399 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.62:35141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":219,"threads_started":1}
I20260810 03:29:34.288049 19604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:29:34.287231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:35141 (local address 127.27.161.1:45397)
0810 03:29:34.287776 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0810 03:29:34.287867 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.62:35141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":282,"threads_started":1}
I20260810 03:29:34.779914 19606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:29:34.779139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:35141 (local address 127.27.161.2:52613)
0810 03:29:34.779645 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0810 03:29:34.779735 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.62:35141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":250,"threads_started":1}
I20260810 03:29:35.265471 19607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:29:35.264645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:35141 (local address 127.27.161.3:36875)
0810 03:29:35.265191 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0810 03:29:35.265283 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.62:35141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":268,"threads_started":1}
I20260810 03:29:35.289875 19609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:29:35.289087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:35141 (local address 127.27.161.1:55651)
0810 03:29:35.289596 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0810 03:29:35.289702 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.62:35141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":261,"threads_started":1}
I20260810 03:29:35.781889 19610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:29:35.781090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:35141 (local address 127.27.161.2:49337)
0810 03:29:35.781609 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0810 03:29:35.781715 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.62:35141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":245,"threads_started":1}
I20260810 03:29:36.267298 19611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:29:36.266487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:35141 (local address 127.27.161.3:50065)
0810 03:29:36.267009 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0810 03:29:36.267125 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.62:35141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":269,"threads_started":1}
I20260810 03:29:36.291957 19614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:29:36.291025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:35141 (local address 127.27.161.1:38997)
0810 03:29:36.291606 (+ 581us) negotiation.cc:107] Waiting for socket to connect
0810 03:29:36.291716 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.62:35141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":250,"threads_started":1}
I20260810 03:29:36.783802 19615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:29:36.782939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:35141 (local address 127.27.161.2:48111)
0810 03:29:36.783493 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0810 03:29:36.783584 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.62:35141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":235,"threads_started":1}
I20260810 03:29:37.269446 19616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:29:37.268390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:35141 (local address 127.27.161.3:52541)
0810 03:29:37.269084 (+ 694us) negotiation.cc:107] Waiting for socket to connect
0810 03:29:37.269209 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.62:35141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":543,"thread_start_us":296,"threads_started":1}
I20260810 03:29:37.293699 19618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:29:37.292953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:35141 (local address 127.27.161.1:59513)
0810 03:29:37.293428 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0810 03:29:37.293530 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.62:35141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":248,"threads_started":1}
I20260810 03:29:37.785640 19624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:29:37.784872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:35141 (local address 127.27.161.2:35113)
0810 03:29:37.785368 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0810 03:29:37.785469 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.62:35141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":247,"threads_started":1}
I20260810 03:29:38.271346 19625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:29:38.270501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:35141 (local address 127.27.161.3:39795)
0810 03:29:38.271002 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0810 03:29:38.271107 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.62:35141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":251,"threads_started":1}
I20260810 03:29:38.295346 19626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:29:38.294593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:35141 (local address 127.27.161.1:47175)
0810 03:29:38.295073 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0810 03:29:38.295160 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.62:35141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":224,"threads_started":1}
I20260810 03:29:38.787500 19627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:29:38.786690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:35141 (local address 127.27.161.2:49855)
0810 03:29:38.787221 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0810 03:29:38.787310 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.62:35141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":253,"threads_started":1}
I20260810 03:29:39.273335 19628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:29:39.272496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:35141 (local address 127.27.161.3:32971)
0810 03:29:39.273032 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0810 03:29:39.273126 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.62:35141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":266,"threads_started":1}
I20260810 03:29:39.297358 19629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:29:39.296513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:35141 (local address 127.27.161.1:56633)
0810 03:29:39.297086 (+ 573us) negotiation.cc:107] Waiting for socket to connect
0810 03:29:39.297184 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.62:35141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":331,"threads_started":1}
I20260810 03:29:39.789397 19630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:29:39.788616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:35141 (local address 127.27.161.2:53643)
0810 03:29:39.789130 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0810 03:29:39.789224 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.62:35141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":279,"threads_started":1}
I20260810 03:29:40.275139 19631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:29:40.274392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:35141 (local address 127.27.161.3:44969)
0810 03:29:40.274882 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0810 03:29:40.274972 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.62:35141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":258,"threads_started":1}
I20260810 03:29:40.299638 19632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:29:40.298635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:35141 (local address 127.27.161.1:55317)
0810 03:29:40.299316 (+ 681us) negotiation.cc:107] Waiting for socket to connect
0810 03:29:40.299431 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.62:35141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":570,"thread_start_us":453,"threads_started":1}
I20260810 03:29:40.791481 19634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:29:40.790531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:35141 (local address 127.27.161.2:42183)
0810 03:29:40.791114 (+ 583us) negotiation.cc:107] Waiting for socket to connect
0810 03:29:40.791247 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.62:35141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":311,"threads_started":1}
I20260810 03:29:41.277014 19635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:29:41.276239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:35141 (local address 127.27.161.3:55167)
0810 03:29:41.276734 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0810 03:29:41.276835 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.62:35141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":258,"threads_started":1}
I20260810 03:29:41.301407 19636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:29:41.300694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:35141 (local address 127.27.161.1:45667)
0810 03:29:41.301161 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0810 03:29:41.301245 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.62:35141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":251,"threads_started":1}
I20260810 03:29:41.793591 19638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:29:41.792581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:35141 (local address 127.27.161.2:39307)
0810 03:29:41.793323 (+ 742us) negotiation.cc:107] Waiting for socket to connect
0810 03:29:41.793424 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.62:35141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":613,"thread_start_us":420,"threads_started":1}
I20260810 03:29:42.278808 19639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:29:42.278102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:35141 (local address 127.27.161.3:35195)
0810 03:29:42.278539 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0810 03:29:42.278645 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.62:35141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":208,"threads_started":1}
I20260810 03:29:42.303438 19641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:29:42.302455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:35141 (local address 127.27.161.1:48471)
0810 03:29:42.303073 (+ 618us) negotiation.cc:107] Waiting for socket to connect
0810 03:29:42.303188 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.62:35141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":319,"threads_started":1}
I20260810 03:29:42.795553 19642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:29:42.794705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:35141 (local address 127.27.161.2:50373)
0810 03:29:42.795250 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0810 03:29:42.795354 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.62:35141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":267,"threads_started":1}
I20260810 03:29:43.280638 19643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:29:43.279809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:35141 (local address 127.27.161.3:46551)
0810 03:29:43.280317 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0810 03:29:43.280420 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.62:35141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":247,"threads_started":1}
I20260810 03:29:43.305275 19645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:29:43.304475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:35141 (local address 127.27.161.1:46265)
0810 03:29:43.304990 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0810 03:29:43.305085 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.62:35141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":267,"threads_started":1}
I20260810 03:29:43.797305 19647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:29:43.796517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:35141 (local address 127.27.161.2:58361)
0810 03:29:43.797022 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0810 03:29:43.797114 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.62:35141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":238,"threads_started":1}
I20260810 03:29:44.282773 19649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:29:44.281829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:35141 (local address 127.27.161.3:51561)
0810 03:29:44.282434 (+ 605us) negotiation.cc:107] Waiting for socket to connect
0810 03:29:44.282535 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.62:35141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":341,"threads_started":1}
I20260810 03:29:44.307334 19650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:29:44.306525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:35141 (local address 127.27.161.1:42517)
0810 03:29:44.307006 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0810 03:29:44.307128 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.62:35141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":244,"threads_started":1}
I20260810 03:29:44.799466 19651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:29:44.798578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:35141 (local address 127.27.161.2:47899)
0810 03:29:44.799095 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0810 03:29:44.799213 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.62:35141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":249,"threads_started":1}
I20260810 03:29:45.260541 28292 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iBH5/build/asan/bin/kudu pid 18881
I20260810 03:29:45.280129 19652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:29:45.261909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38788 (local address 127.27.161.1:37585)
0810 03:29:45.262574 (+ 665us) server_negotiation.cc:206] Beginning negotiation
0810 03:29:45.262594 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0810 03:29:45.262657 (+ 63us) server_negotiation.cc:407] Connection header received
0810 03:29:45.264430 (+ 1773us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:29:45.264441 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:29:45.264885 (+ 444us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:29:45.265187 (+ 302us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:29:45.266673 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:29:45.267908 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:29:45.269142 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:29:45.270154 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:29:45.270294 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:29:45.273056 (+ 2762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:29:45.273065 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:29:45.273086 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:29:45.273137 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:29:45.276266 (+ 3129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:29:45.277089 (+ 823us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:29:45.277097 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:29:45.277104 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:29:45.277301 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:29:45.277715 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:29:45.277720 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:29:45.277722 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:29:45.278372 (+ 650us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:29:45.278711 (+ 339us) server_negotiation.cc:1035] Waiting for connection context
0810 03:29:45.279208 (+ 497us) server_negotiation.cc:299] Negotiation successful
0810 03:29:45.279824 (+ 616us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":462,"thread_start_us":276,"threads_started":1}
I20260810 03:29:45.281589 19151 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:29:45.284907 19654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:29:45.284045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:35141 (local address 127.27.161.3:38485)
0810 03:29:45.284541 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0810 03:29:45.284661 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.62:35141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":247,"threads_started":1}
I20260810 03:29:45.369153 28292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iBH5/build/asan/bin/kudu with pid 18881
I20260810 03:29:45.396706 28292 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iBH5/build/asan/bin/kudu pid 19166
I20260810 03:29:45.415966 19657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:29:45.397792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34726 (local address 127.27.161.2:45581)
0810 03:29:45.398581 (+ 789us) server_negotiation.cc:206] Beginning negotiation
0810 03:29:45.398596 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 03:29:45.398685 (+ 89us) server_negotiation.cc:407] Connection header received
0810 03:29:45.400315 (+ 1630us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:29:45.400325 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:29:45.400775 (+ 450us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:29:45.401130 (+ 355us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:29:45.402808 (+ 1678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:29:45.404141 (+ 1333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:29:45.405438 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:29:45.406423 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:29:45.406610 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:29:45.409375 (+ 2765us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:29:45.409385 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:29:45.409405 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:29:45.409453 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:29:45.412568 (+ 3115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:29:45.413394 (+ 826us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:29:45.413403 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:29:45.413411 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:29:45.413628 (+ 217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:29:45.414013 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:29:45.414018 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:29:45.414021 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:29:45.414605 (+ 584us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:29:45.414837 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0810 03:29:45.415214 (+ 377us) server_negotiation.cc:299] Negotiation successful
0810 03:29:45.415663 (+ 449us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":437,"thread_start_us":259,"threads_started":1}
I20260810 03:29:45.417514 19367 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:29:45.502796 28292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iBH5/build/asan/bin/kudu with pid 19166
I20260810 03:29:45.528491 28292 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iBH5/build/asan/bin/kudu pid 19376
I20260810 03:29:45.547338 19659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:29:45.529627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34240 (local address 127.27.161.3:46513)
0810 03:29:45.530300 (+ 673us) server_negotiation.cc:206] Beginning negotiation
0810 03:29:45.530315 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 03:29:45.530359 (+ 44us) server_negotiation.cc:407] Connection header received
0810 03:29:45.532013 (+ 1654us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:29:45.532024 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:29:45.532476 (+ 452us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:29:45.532783 (+ 307us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:29:45.534463 (+ 1680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:29:45.535734 (+ 1271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:29:45.537015 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:29:45.537706 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:29:45.537847 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:29:45.540653 (+ 2806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:29:45.540663 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:29:45.540682 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:29:45.540728 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:29:45.543780 (+ 3052us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:29:45.544531 (+ 751us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:29:45.544538 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:29:45.544546 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:29:45.544735 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:29:45.545156 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:29:45.545159 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:29:45.545161 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:29:45.545731 (+ 570us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:29:45.546123 (+ 392us) server_negotiation.cc:1035] Waiting for connection context
0810 03:29:45.546568 (+ 445us) server_negotiation.cc:299] Negotiation successful
0810 03:29:45.547012 (+ 444us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":486,"thread_start_us":294,"threads_started":1}
I20260810 03:29:45.548775 19568 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:29:45.634624 28292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iBH5/build/asan/bin/kudu with pid 19376
2026-08-10T03:29:45Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (24853 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 03:29:45 dist-test-slave-vpvm krb5kdc[19667](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 03:29:45 dist-test-slave-vpvm krb5kdc[19667](info): set up 2 sockets
Aug 10 03:29:45 dist-test-slave-vpvm krb5kdc[19667](info): commencing operation
krb5kdc: starting...
W20260810 03:29:46.900167 28292 mini_kdc.cc:121] Time spent starting KDC: real 1.218s user 0.003s sys 0.047s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/krb5kdc/test-user.keytab.
Aug 10 03:29:47 dist-test-slave-vpvm krb5kdc[19667](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332587, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T03:29:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T03:29:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.161.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.161.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.161.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.62.keytab.
Entry for principal kudu/127.27.161.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.62.keytab.
I20260810 03:29:47.225670 28292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iBH5/build/asan/bin/kudu
/tmp/dist-test-task24iBH5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.161.62:34901
--webserver_interface=127.27.161.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.62.keytab
--principal=kudu/127.27.161.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.161.20:42379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.161.62:34901
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:29:47.596881 19687 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:29:47.597218 19687 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:29:47.597362 19687 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:29:47.604945 19687 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260810 03:29:47.605080 19687 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260810 03:29:47.605121 19687 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260810 03:29:47.605194 19687 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:29:47.605273 19687 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:29:47.605360 19687 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:29:47.605429 19687 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:29:47.605509 19687 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:29:47.616091 19687 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.161.20:42379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.27.161.62:34901
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.62.keytab
--principal=kudu/127.27.161.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.161.62:34901
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.161.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:29:47.617517 19687 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:29:47.618914 19687 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:29:47.630769 19697 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:29:47.630851 19700 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:29:47.630873 19698 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:29:47.631764 19687 server_base.cc:1034] running on GCE node
I20260810 03:29:47.632623 19687 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:29:47.634214 19687 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:29:47.635447 19687 hybrid_clock.cc:648] HybridClock initialized: now 1786332587635385 us; error 43 us; skew 500 ppm
I20260810 03:29:47.635807 19687 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:29:47 dist-test-slave-vpvm krb5kdc[19667](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332587, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.161.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:29:47.639230 19687 init.cc:377] Logged in from keytab as kudu/127.27.161.62@KRBTEST.COM (short username kudu)
I20260810 03:29:47.640874 19687 webserver.cc:460] Webserver started at http://127.27.161.62:34355/ using document root <none> and password file <none>
I20260810 03:29:47.641618 19687 fs_manager.cc:359] Metadata directory not provided
I20260810 03:29:47.641731 19687 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:29:47.642073 19687 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:29:47.645412 19687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/minicluster-data/master-0/data/instance:
uuid: "b85f2c98e9de4e5bbb5359e23783aaa0"
format_stamp: "Formatted at 2026-08-10 03:29:47 on dist-test-slave-vpvm"
server_key: "1109bcb983d5b2e9d420ac696de1a21c"
server_key_iv: "d46666874395a5ac58653f3aa33db5cb"
server_key_version: "encryptionkey@0"
I20260810 03:29:47.646308 19687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/minicluster-data/master-0/wal/instance:
uuid: "b85f2c98e9de4e5bbb5359e23783aaa0"
format_stamp: "Formatted at 2026-08-10 03:29:47 on dist-test-slave-vpvm"
server_key: "1109bcb983d5b2e9d420ac696de1a21c"
server_key_iv: "d46666874395a5ac58653f3aa33db5cb"
server_key_version: "encryptionkey@0"
I20260810 03:29:47.652077 19687 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260810 03:29:47.655902 19708 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:29:47.657358 19687 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260810 03:29:47.657527 19687 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/minicluster-data/master-0/data,/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/minicluster-data/master-0/wal
uuid: "b85f2c98e9de4e5bbb5359e23783aaa0"
format_stamp: "Formatted at 2026-08-10 03:29:47 on dist-test-slave-vpvm"
server_key: "1109bcb983d5b2e9d420ac696de1a21c"
server_key_iv: "d46666874395a5ac58653f3aa33db5cb"
server_key_version: "encryptionkey@0"
I20260810 03:29:47.657742 19687 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:29:47.684978 19687 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:29:47.689261 19687 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:29:47.689512 19687 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:29:47.707672 19687 rpc_server.cc:307] RPC server started. Bound to: 127.27.161.62:34901
I20260810 03:29:47.707690 19798 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.161.62:34901 every 8 connection(s)
I20260810 03:29:47.709131 19687 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/minicluster-data/master-0/data/info.pb
I20260810 03:29:47.713091 19799 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:29:47.719609 28292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iBH5/build/asan/bin/kudu as pid 19687
I20260810 03:29:47.719854 28292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/minicluster-data/master-0/wal/instance
I20260810 03:29:47.720728 28292 external_mini_cluster.cc:1428] Setting key 3b239693a9ff98c3fe0a864347cb8836
I20260810 03:29:47.724897 19799 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b85f2c98e9de4e5bbb5359e23783aaa0: Bootstrap starting.
I20260810 03:29:47.728708 19799 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b85f2c98e9de4e5bbb5359e23783aaa0: Neither blocks nor log segments found. Creating new log.
I20260810 03:29:47.729835 19799 log.cc:824] T 00000000000000000000000000000000 P b85f2c98e9de4e5bbb5359e23783aaa0: Log is configured to *not* fsync() on all Append() calls
I20260810 03:29:47.733140 19799 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b85f2c98e9de4e5bbb5359e23783aaa0: No bootstrap required, opened a new log
Aug 10 03:29:47 dist-test-slave-vpvm krb5kdc[19667](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332587, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.161.62@KRBTEST.COM
I20260810 03:29:47.742151 19799 raft_consensus.cc:348] T 00000000000000000000000000000000 P b85f2c98e9de4e5bbb5359e23783aaa0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b85f2c98e9de4e5bbb5359e23783aaa0" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 34901 } }
I20260810 03:29:47.742586 19799 raft_consensus.cc:374] T 00000000000000000000000000000000 P b85f2c98e9de4e5bbb5359e23783aaa0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:29:47.742689 19799 raft_consensus.cc:729] T 00000000000000000000000000000000 P b85f2c98e9de4e5bbb5359e23783aaa0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b85f2c98e9de4e5bbb5359e23783aaa0, State: Initialized, Role: FOLLOWER
I20260810 03:29:47.743374 19799 consensus_queue.cc:260] T 00000000000000000000000000000000 P b85f2c98e9de4e5bbb5359e23783aaa0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b85f2c98e9de4e5bbb5359e23783aaa0" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 34901 } }
I20260810 03:29:47.743664 19799 raft_consensus.cc:388] T 00000000000000000000000000000000 P b85f2c98e9de4e5bbb5359e23783aaa0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 03:29:47.743836 19799 raft_consensus.cc:482] T 00000000000000000000000000000000 P b85f2c98e9de4e5bbb5359e23783aaa0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 03:29:47.744063 19799 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b85f2c98e9de4e5bbb5359e23783aaa0 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:29:47.745277 19805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:29:47.723059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41106 (local address 127.27.161.62:34901)
0810 03:29:47.723882 (+ 823us) server_negotiation.cc:206] Beginning negotiation
0810 03:29:47.723895 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 03:29:47.723942 (+ 47us) server_negotiation.cc:407] Connection header received
0810 03:29:47.725824 (+ 1882us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:29:47.725859 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:29:47.726327 (+ 468us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:29:47.726694 (+ 367us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:29:47.728551 (+ 1857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:29:47.729972 (+ 1421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:29:47.731732 (+ 1760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:29:47.732413 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:29:47.732545 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:29:47.737445 (+ 4900us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:29:47.737454 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:29:47.737472 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:29:47.737513 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:29:47.740726 (+ 3213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:29:47.741503 (+ 777us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:29:47.741511 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:29:47.741518 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:29:47.741704 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:29:47.742161 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:29:47.742164 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:29:47.742167 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:29:47.742828 (+ 661us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:29:47.743155 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0810 03:29:47.743627 (+ 472us) server_negotiation.cc:299] Negotiation successful
0810 03:29:47.744112 (+ 485us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":606,"thread_start_us":357,"threads_started":1}
I20260810 03:29:47.746742 19799 raft_consensus.cc:504] T 00000000000000000000000000000000 P b85f2c98e9de4e5bbb5359e23783aaa0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b85f2c98e9de4e5bbb5359e23783aaa0" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 34901 } }
I20260810 03:29:47.747371 19799 leader_election.cc:302] T 00000000000000000000000000000000 P b85f2c98e9de4e5bbb5359e23783aaa0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b85f2c98e9de4e5bbb5359e23783aaa0; no voters:
I20260810 03:29:47.748078 19799 leader_election.cc:288] T 00000000000000000000000000000000 P b85f2c98e9de4e5bbb5359e23783aaa0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 03:29:47.748196 19807 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b85f2c98e9de4e5bbb5359e23783aaa0 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:29:47.749234 19807 raft_consensus.cc:686] T 00000000000000000000000000000000 P b85f2c98e9de4e5bbb5359e23783aaa0 [term 1 LEADER]: Becoming Leader. State: Replica: b85f2c98e9de4e5bbb5359e23783aaa0, State: Running, Role: LEADER
I20260810 03:29:47.749765 19807 consensus_queue.cc:237] T 00000000000000000000000000000000 P b85f2c98e9de4e5bbb5359e23783aaa0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b85f2c98e9de4e5bbb5359e23783aaa0" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 34901 } }
I20260810 03:29:47.750293 19799 sys_catalog.cc:564] T 00000000000000000000000000000000 P b85f2c98e9de4e5bbb5359e23783aaa0 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:29:47.752430 19808 sys_catalog.cc:455] T 00000000000000000000000000000000 P b85f2c98e9de4e5bbb5359e23783aaa0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b85f2c98e9de4e5bbb5359e23783aaa0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b85f2c98e9de4e5bbb5359e23783aaa0" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 34901 } } }
I20260810 03:29:47.752836 19808 sys_catalog.cc:458] T 00000000000000000000000000000000 P b85f2c98e9de4e5bbb5359e23783aaa0 [sys.catalog]: This master's current role is: LEADER
I20260810 03:29:47.753175 19810 sys_catalog.cc:455] T 00000000000000000000000000000000 P b85f2c98e9de4e5bbb5359e23783aaa0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b85f2c98e9de4e5bbb5359e23783aaa0. Latest consensus state: current_term: 1 leader_uuid: "b85f2c98e9de4e5bbb5359e23783aaa0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b85f2c98e9de4e5bbb5359e23783aaa0" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 34901 } } }
I20260810 03:29:47.753511 19810 sys_catalog.cc:458] T 00000000000000000000000000000000 P b85f2c98e9de4e5bbb5359e23783aaa0 [sys.catalog]: This master's current role is: LEADER
I20260810 03:29:47.755240 19814 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 03:29:47.760497 19814 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 03:29:47.770690 19814 catalog_manager.cc:1347] Generated new cluster ID: 49064dc1c4064843a5cd4d7e37ffd0d9
I20260810 03:29:47.770818 19814 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 03:29:47.783608 19814 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 03:29:47.784973 19814 catalog_manager.cc:1504] Loading token signing keys...
I20260810 03:29:47.791882 19814 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b85f2c98e9de4e5bbb5359e23783aaa0: Generated new TSK 0
I20260810 03:29:47.792755 19814 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.161.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.161.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.161.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.1.keytab.
Entry for principal kudu/127.27.161.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.1.keytab.
I20260810 03:29:47.883652 28292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iBH5/build/asan/bin/kudu
/tmp/dist-test-task24iBH5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.161.1:0
--local_ip_for_outbound_sockets=127.27.161.1
--webserver_interface=127.27.161.1
--webserver_port=0
--tserver_master_addrs=127.27.161.62:34901
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.1.keytab
--principal=kudu/127.27.161.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.161.20:42379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:29:48.265290 19842 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:29:48.265597 19842 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:29:48.265713 19842 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:29:48.273443 19842 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:29:48.273581 19842 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:29:48.273715 19842 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.161.1
I20260810 03:29:48.283725 19842 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.161.20:42379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.1.keytab
--principal=kudu/127.27.161.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.161.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.161.1
--webserver_port=0
--tserver_master_addrs=127.27.161.62:34901
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.161.1
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:29:48.285049 19842 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:29:48.286384 19842 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:29:48.300096 19849 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:29:48.300190 19850 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:29:48.300136 19852 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:29:48.303550 19842 server_base.cc:1034] running on GCE node
I20260810 03:29:48.304235 19842 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:29:48.305292 19842 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:29:48.306519 19842 hybrid_clock.cc:648] HybridClock initialized: now 1786332588306451 us; error 60 us; skew 500 ppm
I20260810 03:29:48.306842 19842 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:29:48 dist-test-slave-vpvm krb5kdc[19667](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332588, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.161.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:29:48.310057 19842 init.cc:377] Logged in from keytab as kudu/127.27.161.1@KRBTEST.COM (short username kudu)
I20260810 03:29:48.311551 19842 webserver.cc:460] Webserver started at http://127.27.161.1:43831/ using document root <none> and password file <none>
I20260810 03:29:48.312265 19842 fs_manager.cc:359] Metadata directory not provided
I20260810 03:29:48.312366 19842 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:29:48.312633 19842 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:29:48.315649 19842 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/minicluster-data/ts-0/data/instance:
uuid: "fac5507619954c158ebdd47a7d73fdc6"
format_stamp: "Formatted at 2026-08-10 03:29:48 on dist-test-slave-vpvm"
server_key: "a5851e1672c45620019b53ae23d94ea1"
server_key_iv: "ee5f60108cbc24f6eda769236ae9dc3e"
server_key_version: "encryptionkey@0"
I20260810 03:29:48.316424 19842 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/minicluster-data/ts-0/wal/instance:
uuid: "fac5507619954c158ebdd47a7d73fdc6"
format_stamp: "Formatted at 2026-08-10 03:29:48 on dist-test-slave-vpvm"
server_key: "a5851e1672c45620019b53ae23d94ea1"
server_key_iv: "ee5f60108cbc24f6eda769236ae9dc3e"
server_key_version: "encryptionkey@0"
I20260810 03:29:48.322225 19842 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260810 03:29:48.325940 19865 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:29:48.327379 19842 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 03:29:48.327536 19842 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/minicluster-data/ts-0/data,/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/minicluster-data/ts-0/wal
uuid: "fac5507619954c158ebdd47a7d73fdc6"
format_stamp: "Formatted at 2026-08-10 03:29:48 on dist-test-slave-vpvm"
server_key: "a5851e1672c45620019b53ae23d94ea1"
server_key_iv: "ee5f60108cbc24f6eda769236ae9dc3e"
server_key_version: "encryptionkey@0"
I20260810 03:29:48.327757 19842 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:29:48.344084 19842 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:29:48.348299 19842 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:29:48.348583 19842 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:29:48.349612 19842 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:29:48.351105 19842 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:29:48.351189 19842 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:29:48.351328 19842 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:29:48.351383 19842 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:29:48.383530 19842 rpc_server.cc:307] RPC server started. Bound to: 127.27.161.1:38339
I20260810 03:29:48.383580 20030 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.161.1:38339 every 8 connection(s)
I20260810 03:29:48.384840 19842 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/minicluster-data/ts-0/data/info.pb
I20260810 03:29:48.387894 28292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iBH5/build/asan/bin/kudu as pid 19842
I20260810 03:29:48.388131 28292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/minicluster-data/ts-0/wal/instance
I20260810 03:29:48.388863 28292 external_mini_cluster.cc:1428] Setting key 8faf343c58ee7c0a2bb1798409f3648b
Aug 10 03:29:48 dist-test-slave-vpvm krb5kdc[19667](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332588, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.161.1@KRBTEST.COM for kudu/127.27.161.62@KRBTEST.COM
I20260810 03:29:48.406246 20033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:29:48.387770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.161.1:36641 (local address 127.27.161.62:34901)
0810 03:29:48.388416 (+ 646us) server_negotiation.cc:206] Beginning negotiation
0810 03:29:48.388423 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:29:48.389634 (+ 1211us) server_negotiation.cc:407] Connection header received
0810 03:29:48.391636 (+ 2002us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:29:48.391642 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:29:48.391725 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:29:48.391831 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:29:48.393376 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:29:48.394353 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:29:48.395941 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:29:48.396475 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:29:48.396613 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:29:48.400068 (+ 3455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:29:48.400075 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:29:48.400079 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:29:48.400119 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:29:48.402999 (+ 2880us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:29:48.403813 (+ 814us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:29:48.403818 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:29:48.403821 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:29:48.403964 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:29:48.404356 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:29:48.404361 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:29:48.404363 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:29:48.404684 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:29:48.404798 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0810 03:29:48.405745 (+ 947us) server_negotiation.cc:299] Negotiation successful
0810 03:29:48.405925 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":493,"thread_start_us":343,"threads_started":1}
I20260810 03:29:48.407737 20034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:29:48.388398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:34901 (local address 127.27.161.1:36641)
0810 03:29:48.389364 (+ 966us) negotiation.cc:107] Waiting for socket to connect
0810 03:29:48.389425 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0810 03:29:48.391282 (+ 1857us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:29:48.392042 (+ 760us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:29:48.392051 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:29:48.392598 (+ 547us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:29:48.393200 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:29:48.393219 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:29:48.394510 (+ 1291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:29:48.394515 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:29:48.395794 (+ 1279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:29:48.395807 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:29:48.396671 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:29:48.396676 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:29:48.396876 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:29:48.397505 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:29:48.397528 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:29:48.399843 (+ 2315us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:29:48.403154 (+ 3311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:29:48.403162 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:29:48.403174 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:29:48.403690 (+ 516us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:29:48.404061 (+ 371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:29:48.404064 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:29:48.404066 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:29:48.404260 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:29:48.404838 (+ 578us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:29:48.404847 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:29:48.405306 (+ 459us) client_negotiation.cc:769] Sending connection context
0810 03:29:48.405637 (+ 331us) client_negotiation.cc:240] Negotiation successful
0810 03:29:48.406239 (+ 602us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":659,"thread_start_us":430,"threads_started":1}
I20260810 03:29:48.409392 20031 heartbeater.cc:347] Connected to a master server at 127.27.161.62:34901
I20260810 03:29:48.409718 20031 heartbeater.cc:464] Registering TS with master...
I20260810 03:29:48.410506 20031 heartbeater.cc:511] Master 127.27.161.62:34901 requested a full tablet report, sending...
I20260810 03:29:48.412978 19734 ts_manager.cc:194] Registered new tserver with Master: fac5507619954c158ebdd47a7d73fdc6 (127.27.161.1:38339)
I20260810 03:29:48.414973 19734 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.161.1@KRBTEST.COM'} at 127.27.161.1:36641
WARNING: no policy specified for kudu/127.27.161.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.161.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.161.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.2.keytab.
Entry for principal kudu/127.27.161.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.2.keytab.
I20260810 03:29:48.462620 28292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iBH5/build/asan/bin/kudu
/tmp/dist-test-task24iBH5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.161.2:0
--local_ip_for_outbound_sockets=127.27.161.2
--webserver_interface=127.27.161.2
--webserver_port=0
--tserver_master_addrs=127.27.161.62:34901
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.2.keytab
--principal=kudu/127.27.161.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.161.20:42379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:29:48.812386 20038 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:29:48.812726 20038 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:29:48.812871 20038 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:29:48.820320 20038 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:29:48.820469 20038 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:29:48.820623 20038 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.161.2
I20260810 03:29:48.831067 20038 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.161.20:42379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.2.keytab
--principal=kudu/127.27.161.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.161.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.161.2
--webserver_port=0
--tserver_master_addrs=127.27.161.62:34901
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.161.2
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:29:48.832571 20038 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:29:48.834008 20038 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:29:48.847914 20043 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:29:48.847890 20047 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:29:48.847858 20045 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:29:48.848834 20038 server_base.cc:1034] running on GCE node
I20260810 03:29:48.849449 20038 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:29:48.850536 20038 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:29:48.851763 20038 hybrid_clock.cc:648] HybridClock initialized: now 1786332588851701 us; error 52 us; skew 500 ppm
I20260810 03:29:48.852113 20038 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:29:48 dist-test-slave-vpvm krb5kdc[19667](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332588, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.161.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:29:48.855563 20038 init.cc:377] Logged in from keytab as kudu/127.27.161.2@KRBTEST.COM (short username kudu)
I20260810 03:29:48.857369 20038 webserver.cc:460] Webserver started at http://127.27.161.2:43657/ using document root <none> and password file <none>
I20260810 03:29:48.858263 20038 fs_manager.cc:359] Metadata directory not provided
I20260810 03:29:48.858415 20038 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:29:48.858783 20038 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:29:48.862398 20038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/minicluster-data/ts-1/data/instance:
uuid: "f01667cb33c543839000d6eb9fb758d4"
format_stamp: "Formatted at 2026-08-10 03:29:48 on dist-test-slave-vpvm"
server_key: "28983bd267833e90f784d217c84c1e21"
server_key_iv: "442c9269eed320d80b27b8851ba899c6"
server_key_version: "encryptionkey@0"
I20260810 03:29:48.863456 20038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/minicluster-data/ts-1/wal/instance:
uuid: "f01667cb33c543839000d6eb9fb758d4"
format_stamp: "Formatted at 2026-08-10 03:29:48 on dist-test-slave-vpvm"
server_key: "28983bd267833e90f784d217c84c1e21"
server_key_iv: "442c9269eed320d80b27b8851ba899c6"
server_key_version: "encryptionkey@0"
I20260810 03:29:48.869692 20038 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260810 03:29:48.874414 20056 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:29:48.875974 20038 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260810 03:29:48.876178 20038 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/minicluster-data/ts-1/data,/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/minicluster-data/ts-1/wal
uuid: "f01667cb33c543839000d6eb9fb758d4"
format_stamp: "Formatted at 2026-08-10 03:29:48 on dist-test-slave-vpvm"
server_key: "28983bd267833e90f784d217c84c1e21"
server_key_iv: "442c9269eed320d80b27b8851ba899c6"
server_key_version: "encryptionkey@0"
I20260810 03:29:48.876394 20038 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:29:48.917788 20038 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:29:48.922247 20038 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:29:48.922559 20038 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:29:48.923650 20038 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:29:48.925096 20038 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:29:48.925182 20038 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:29:48.925308 20038 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:29:48.925364 20038 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:29:48.959421 20038 rpc_server.cc:307] RPC server started. Bound to: 127.27.161.2:36973
I20260810 03:29:48.959452 20225 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.161.2:36973 every 8 connection(s)
I20260810 03:29:48.961119 20038 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/minicluster-data/ts-1/data/info.pb
I20260810 03:29:48.965790 28292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iBH5/build/asan/bin/kudu as pid 20038
I20260810 03:29:48.966028 28292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/minicluster-data/ts-1/wal/instance
I20260810 03:29:48.966656 28292 external_mini_cluster.cc:1428] Setting key 02b211f84da914baddaef83de266340b
Aug 10 03:29:48 dist-test-slave-vpvm krb5kdc[19667](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332588, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.161.2@KRBTEST.COM for kudu/127.27.161.62@KRBTEST.COM
I20260810 03:29:48.985085 20233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:29:48.963982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.161.2:42905 (local address 127.27.161.62:34901)
0810 03:29:48.964478 (+ 496us) server_negotiation.cc:206] Beginning negotiation
0810 03:29:48.964485 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:29:48.965542 (+ 1057us) server_negotiation.cc:407] Connection header received
0810 03:29:48.968009 (+ 2467us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:29:48.968013 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:29:48.968086 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:29:48.968180 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:29:48.970380 (+ 2200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:29:48.971152 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:29:48.973691 (+ 2539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:29:48.974256 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:29:48.974375 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:29:48.978926 (+ 4551us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:29:48.978932 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:29:48.978936 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:29:48.978984 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:29:48.981735 (+ 2751us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:29:48.982601 (+ 866us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:29:48.982608 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:29:48.982610 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:29:48.982754 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:29:48.983234 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:29:48.983239 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:29:48.983241 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:29:48.983585 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:29:48.983743 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0810 03:29:48.984526 (+ 783us) server_negotiation.cc:299] Negotiation successful
0810 03:29:48.984779 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":356,"thread_start_us":223,"threads_started":1}
I20260810 03:29:48.986006 20235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:29:48.964578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:34901 (local address 127.27.161.2:42905)
0810 03:29:48.965345 (+ 767us) negotiation.cc:107] Waiting for socket to connect
0810 03:29:48.965396 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0810 03:29:48.967704 (+ 2308us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:29:48.968474 (+ 770us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:29:48.968487 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:29:48.969191 (+ 704us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:29:48.970154 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:29:48.970183 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:29:48.971331 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:29:48.971337 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:29:48.973381 (+ 2044us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:29:48.973400 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:29:48.974526 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:29:48.974534 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:29:48.974844 (+ 310us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:29:48.975672 (+ 828us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:29:48.975704 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:29:48.978612 (+ 2908us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:29:48.981913 (+ 3301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:29:48.981923 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:29:48.981938 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:29:48.982452 (+ 514us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:29:48.982873 (+ 421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:29:48.982879 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:29:48.982883 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:29:48.983094 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:29:48.983742 (+ 648us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:29:48.983749 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:29:48.984113 (+ 364us) client_negotiation.cc:769] Sending connection context
0810 03:29:48.984379 (+ 266us) client_negotiation.cc:240] Negotiation successful
0810 03:29:48.984819 (+ 440us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":500,"thread_start_us":283,"threads_started":1}
I20260810 03:29:48.987715 20230 heartbeater.cc:347] Connected to a master server at 127.27.161.62:34901
I20260810 03:29:48.988041 20230 heartbeater.cc:464] Registering TS with master...
I20260810 03:29:48.988826 20230 heartbeater.cc:511] Master 127.27.161.62:34901 requested a full tablet report, sending...
I20260810 03:29:48.990464 19734 ts_manager.cc:194] Registered new tserver with Master: f01667cb33c543839000d6eb9fb758d4 (127.27.161.2:36973)
I20260810 03:29:48.991333 19734 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.161.2@KRBTEST.COM'} at 127.27.161.2:42905
WARNING: no policy specified for kudu/127.27.161.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.161.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.161.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.3.keytab.
Entry for principal kudu/127.27.161.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.3.keytab.
I20260810 03:29:49.041589 28292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iBH5/build/asan/bin/kudu
/tmp/dist-test-task24iBH5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.161.3:0
--local_ip_for_outbound_sockets=127.27.161.3
--webserver_interface=127.27.161.3
--webserver_port=0
--tserver_master_addrs=127.27.161.62:34901
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.3.keytab
--principal=kudu/127.27.161.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.161.20:42379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:29:49.396349 20239 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:29:49.396641 20239 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:29:49.396781 20239 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:29:49.404274 20239 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:29:49.404420 20239 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:29:49.404583 20239 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.161.3
I20260810 03:29:49.414812 20239 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.161.20:42379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.3.keytab
--principal=kudu/127.27.161.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.161.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.161.3
--webserver_port=0
--tserver_master_addrs=127.27.161.62:34901
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.161.3
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:29:49.416154 20239 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:29:49.417532 20239 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 03:29:49.418238 20031 heartbeater.cc:503] Master 127.27.161.62:34901 was elected leader, sending a full tablet report...
W20260810 03:29:49.431553 20246 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:29:49.431592 20247 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:29:49.431633 20250 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:29:49.432524 20239 server_base.cc:1034] running on GCE node
I20260810 03:29:49.433136 20239 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:29:49.434238 20239 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:29:49.435444 20239 hybrid_clock.cc:648] HybridClock initialized: now 1786332589435394 us; error 45 us; skew 500 ppm
I20260810 03:29:49.435766 20239 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:29:49 dist-test-slave-vpvm krb5kdc[19667](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332589, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.161.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:29:49.438931 20239 init.cc:377] Logged in from keytab as kudu/127.27.161.3@KRBTEST.COM (short username kudu)
I20260810 03:29:49.440521 20239 webserver.cc:460] Webserver started at http://127.27.161.3:42135/ using document root <none> and password file <none>
I20260810 03:29:49.441296 20239 fs_manager.cc:359] Metadata directory not provided
I20260810 03:29:49.441421 20239 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:29:49.441744 20239 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:29:49.445003 20239 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/minicluster-data/ts-2/data/instance:
uuid: "c9a9fd5ce6b04fd5b5d2480a800f403c"
format_stamp: "Formatted at 2026-08-10 03:29:49 on dist-test-slave-vpvm"
server_key: "eed21fa6d563145c0f2de57944d4bf26"
server_key_iv: "c70ac530c2efd77a7ba72de0ce6a5193"
server_key_version: "encryptionkey@0"
I20260810 03:29:49.445798 20239 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/minicluster-data/ts-2/wal/instance:
uuid: "c9a9fd5ce6b04fd5b5d2480a800f403c"
format_stamp: "Formatted at 2026-08-10 03:29:49 on dist-test-slave-vpvm"
server_key: "eed21fa6d563145c0f2de57944d4bf26"
server_key_iv: "c70ac530c2efd77a7ba72de0ce6a5193"
server_key_version: "encryptionkey@0"
I20260810 03:29:49.451373 20239 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260810 03:29:49.455179 20261 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:29:49.456568 20239 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 03:29:49.456741 20239 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/minicluster-data/ts-2/data,/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/minicluster-data/ts-2/wal
uuid: "c9a9fd5ce6b04fd5b5d2480a800f403c"
format_stamp: "Formatted at 2026-08-10 03:29:49 on dist-test-slave-vpvm"
server_key: "eed21fa6d563145c0f2de57944d4bf26"
server_key_iv: "c70ac530c2efd77a7ba72de0ce6a5193"
server_key_version: "encryptionkey@0"
I20260810 03:29:49.456950 20239 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:29:49.480115 20239 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:29:49.484211 20239 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:29:49.484489 20239 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:29:49.485508 20239 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:29:49.486927 20239 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:29:49.487008 20239 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:29:49.487099 20239 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:29:49.487141 20239 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:29:49.518193 20239 rpc_server.cc:307] RPC server started. Bound to: 127.27.161.3:44627
I20260810 03:29:49.518234 20437 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.161.3:44627 every 8 connection(s)
I20260810 03:29:49.519479 20239 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/minicluster-data/ts-2/data/info.pb
I20260810 03:29:49.524243 28292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iBH5/build/asan/bin/kudu as pid 20239
I20260810 03:29:49.524410 28292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/minicluster-data/ts-2/wal/instance
I20260810 03:29:49.524878 28292 external_mini_cluster.cc:1428] Setting key c4f8358cff493e762507cf536efe950c
Aug 10 03:29:49 dist-test-slave-vpvm krb5kdc[19667](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332589, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.161.3@KRBTEST.COM for kudu/127.27.161.62@KRBTEST.COM
I20260810 03:29:49.537828 20441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:29:49.521914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.161.3:34783 (local address 127.27.161.62:34901)
0810 03:29:49.522368 (+ 454us) server_negotiation.cc:206] Beginning negotiation
0810 03:29:49.522373 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:29:49.523253 (+ 880us) server_negotiation.cc:407] Connection header received
0810 03:29:49.525138 (+ 1885us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:29:49.525144 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:29:49.525240 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:29:49.525336 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:29:49.526909 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:29:49.527790 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:29:49.529248 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:29:49.529614 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:29:49.529710 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:29:49.532904 (+ 3194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:29:49.532909 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:29:49.532913 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:29:49.532954 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:29:49.535384 (+ 2430us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:29:49.536007 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:29:49.536011 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:29:49.536013 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:29:49.536123 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:29:49.536442 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:29:49.536445 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:29:49.536447 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:29:49.536692 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:29:49.536816 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0810 03:29:49.537388 (+ 572us) server_negotiation.cc:299] Negotiation successful
0810 03:29:49.537545 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":346,"thread_start_us":223,"threads_started":1}
I20260810 03:29:49.538727 20443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:29:49.522402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:34901 (local address 127.27.161.3:34783)
0810 03:29:49.523090 (+ 688us) negotiation.cc:107] Waiting for socket to connect
0810 03:29:49.523132 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0810 03:29:49.524873 (+ 1741us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:29:49.525568 (+ 695us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:29:49.525576 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:29:49.526212 (+ 636us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:29:49.526758 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:29:49.526777 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:29:49.527925 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:29:49.527929 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:29:49.529137 (+ 1208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:29:49.529150 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:29:49.529775 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:29:49.529780 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:29:49.529994 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:29:49.530604 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:29:49.530627 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:29:49.532710 (+ 2083us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:29:49.535503 (+ 2793us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:29:49.535509 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:29:49.535521 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:29:49.535918 (+ 397us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:29:49.536194 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:29:49.536197 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:29:49.536199 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:29:49.536364 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:29:49.536788 (+ 424us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:29:49.536794 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:29:49.537093 (+ 299us) client_negotiation.cc:769] Sending connection context
0810 03:29:49.537306 (+ 213us) client_negotiation.cc:240] Negotiation successful
0810 03:29:49.537668 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":256,"threads_started":1}
I20260810 03:29:49.540086 20438 heartbeater.cc:347] Connected to a master server at 127.27.161.62:34901
I20260810 03:29:49.540341 20438 heartbeater.cc:464] Registering TS with master...
I20260810 03:29:49.540987 20438 heartbeater.cc:511] Master 127.27.161.62:34901 requested a full tablet report, sending...
I20260810 03:29:49.542429 19734 ts_manager.cc:194] Registered new tserver with Master: c9a9fd5ce6b04fd5b5d2480a800f403c (127.27.161.3:44627)
I20260810 03:29:49.543182 19734 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.161.3@KRBTEST.COM'} at 127.27.161.3:34783
I20260810 03:29:49.551966 28292 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 03:29:49.568373 20441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:29:49.556350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53818 (local address 127.27.161.62:34901)
0810 03:29:49.556508 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0810 03:29:49.556512 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:29:49.557003 (+ 491us) server_negotiation.cc:407] Connection header received
0810 03:29:49.557161 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:29:49.557164 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:29:49.557241 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:29:49.557319 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:29:49.558808 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:29:49.559521 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:29:49.560851 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:29:49.561350 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:29:49.561442 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:29:49.563805 (+ 2363us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:29:49.563810 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:29:49.563813 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:29:49.563844 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:29:49.566305 (+ 2461us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:29:49.566897 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:29:49.566901 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:29:49.566903 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:29:49.567009 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:29:49.567285 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:29:49.567289 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:29:49.567290 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:29:49.567582 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:29:49.567756 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0810 03:29:49.567976 (+ 220us) server_negotiation.cc:299] Negotiation successful
0810 03:29:49.568150 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260810 03:29:49.572289 19734 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53818:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 03:29:49.579257 19734 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 03:29:49.618425 20460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:29:49.592137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53128 (local address 127.27.161.3:44627)
0810 03:29:49.604625 (+ 12488us) server_negotiation.cc:206] Beginning negotiation
0810 03:29:49.604635 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 03:29:49.609404 (+ 4769us) server_negotiation.cc:407] Connection header received
0810 03:29:49.610422 (+ 1018us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:29:49.610430 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:29:49.610691 (+ 261us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:29:49.610854 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:29:49.612172 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:29:49.613694 (+ 1522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:29:49.616059 (+ 2365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:29:49.617248 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:29:49.617361 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:29:49.617622 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0810 03:29:49.617884 (+ 262us) server_negotiation.cc:299] Negotiation successful
0810 03:29:49.618082 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12300,"thread_start_us":12125,"threads_started":1}
I20260810 03:29:49.618458 20461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:29:49.591822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.3:44627 (local address 127.0.0.1:53128)
0810 03:29:49.609201 (+ 17379us) negotiation.cc:107] Waiting for socket to connect
0810 03:29:49.609244 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0810 03:29:49.610120 (+ 876us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:29:49.611183 (+ 1063us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:29:49.611186 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:29:49.611207 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:29:49.611883 (+ 676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:29:49.611897 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:29:49.613837 (+ 1940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:29:49.613842 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:29:49.615906 (+ 2064us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:29:49.615919 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:29:49.617446 (+ 1527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:29:49.617452 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:29:49.617651 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:29:49.617708 (+ 57us) client_negotiation.cc:769] Sending connection context
0810 03:29:49.617823 (+ 115us) client_negotiation.cc:240] Negotiation successful
0810 03:29:49.618035 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17093,"mutex_wait_us":287,"spinlock_wait_cycles":5376,"thread_start_us":17866,"threads_started":1}
I20260810 03:29:49.620345 20458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:29:49.591951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.2:36973 (local address 127.0.0.1:34098)
0810 03:29:49.609890 (+ 17939us) negotiation.cc:107] Waiting for socket to connect
0810 03:29:49.609908 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 03:29:49.610130 (+ 222us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:29:49.611144 (+ 1014us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:29:49.611152 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:29:49.611191 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:29:49.611834 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:29:49.611852 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:29:49.613726 (+ 1874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:29:49.613731 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:29:49.616715 (+ 2984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:29:49.616730 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:29:49.619362 (+ 2632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:29:49.619369 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:29:49.619595 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:29:49.619634 (+ 39us) client_negotiation.cc:769] Sending connection context
0810 03:29:49.619733 (+ 99us) client_negotiation.cc:240] Negotiation successful
0810 03:29:49.619873 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17803,"mutex_wait_us":236,"spinlock_wait_cycles":20992,"thread_start_us":17644,"threads_started":1}
I20260810 03:29:49.620592 20463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:29:49.593822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39654 (local address 127.27.161.1:38339)
0810 03:29:49.602455 (+ 8633us) server_negotiation.cc:206] Beginning negotiation
0810 03:29:49.602468 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 03:29:49.610088 (+ 7620us) server_negotiation.cc:407] Connection header received
0810 03:29:49.610392 (+ 304us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:29:49.610401 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:29:49.610683 (+ 282us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:29:49.610844 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:29:49.612012 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:29:49.613196 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:29:49.616929 (+ 3733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:29:49.618944 (+ 2015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:29:49.619075 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:29:49.619318 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0810 03:29:49.620047 (+ 729us) server_negotiation.cc:299] Negotiation successful
0810 03:29:49.620235 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8405,"thread_start_us":8190,"threads_started":1}
I20260810 03:29:49.621152 20459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:29:49.592243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34098 (local address 127.27.161.2:36973)
0810 03:29:49.608231 (+ 15988us) server_negotiation.cc:206] Beginning negotiation
0810 03:29:49.608287 (+ 56us) server_negotiation.cc:399] Waiting for connection header
0810 03:29:49.610335 (+ 2048us) server_negotiation.cc:407] Connection header received
0810 03:29:49.610432 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:29:49.610437 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:29:49.610669 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:29:49.610851 (+ 182us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:29:49.612042 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:29:49.613577 (+ 1535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:29:49.616932 (+ 3355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:29:49.618489 (+ 1557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:29:49.618985 (+ 496us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:29:49.619206 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0810 03:29:49.620628 (+ 1422us) server_negotiation.cc:299] Negotiation successful
0810 03:29:49.620824 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15698,"thread_start_us":16073,"threads_started":1}
I20260810 03:29:49.621461 20457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:29:49.593605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.1:38339 (local address 127.0.0.1:39654)
0810 03:29:49.609942 (+ 16337us) negotiation.cc:107] Waiting for socket to connect
0810 03:29:49.609976 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0810 03:29:49.610184 (+ 208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:29:49.611151 (+ 967us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:29:49.611155 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:29:49.611195 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:29:49.611794 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:29:49.611808 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:29:49.613354 (+ 1546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:29:49.613359 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:29:49.616713 (+ 3354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:29:49.616731 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:29:49.619513 (+ 2782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:29:49.619520 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:29:49.619691 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:29:49.619725 (+ 34us) client_negotiation.cc:769] Sending connection context
0810 03:29:49.620884 (+ 1159us) client_negotiation.cc:240] Negotiation successful
0810 03:29:49.621039 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16232,"mutex_wait_us":251,"spinlock_wait_cycles":3072,"thread_start_us":15146,"threads_started":1}
I20260810 03:29:49.625321 20139 tablet_service.cc:1467] Processing CreateTablet for tablet 5e8b86f0562049a2a19d33f574b0da93 (DEFAULT_TABLE table=test-table [id=a88440a4cc834973a1247345a76b6fdc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:29:49.625324 19936 tablet_service.cc:1467] Processing CreateTablet for tablet 5e8b86f0562049a2a19d33f574b0da93 (DEFAULT_TABLE table=test-table [id=a88440a4cc834973a1247345a76b6fdc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:29:49.626850 20139 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e8b86f0562049a2a19d33f574b0da93. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:29:49.626852 19936 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e8b86f0562049a2a19d33f574b0da93. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:29:49.622519 20340 tablet_service.cc:1467] Processing CreateTablet for tablet 5e8b86f0562049a2a19d33f574b0da93 (DEFAULT_TABLE table=test-table [id=a88440a4cc834973a1247345a76b6fdc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:29:49.629876 20340 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e8b86f0562049a2a19d33f574b0da93. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:29:49.640991 20465 tablet_bootstrap.cc:492] T 5e8b86f0562049a2a19d33f574b0da93 P c9a9fd5ce6b04fd5b5d2480a800f403c: Bootstrap starting.
I20260810 03:29:49.642256 20467 tablet_bootstrap.cc:492] T 5e8b86f0562049a2a19d33f574b0da93 P f01667cb33c543839000d6eb9fb758d4: Bootstrap starting.
I20260810 03:29:49.646029 20465 tablet_bootstrap.cc:654] T 5e8b86f0562049a2a19d33f574b0da93 P c9a9fd5ce6b04fd5b5d2480a800f403c: Neither blocks nor log segments found. Creating new log.
I20260810 03:29:49.646029 20467 tablet_bootstrap.cc:654] T 5e8b86f0562049a2a19d33f574b0da93 P f01667cb33c543839000d6eb9fb758d4: Neither blocks nor log segments found. Creating new log.
I20260810 03:29:49.646680 20466 tablet_bootstrap.cc:492] T 5e8b86f0562049a2a19d33f574b0da93 P fac5507619954c158ebdd47a7d73fdc6: Bootstrap starting.
I20260810 03:29:49.647395 20465 log.cc:824] T 5e8b86f0562049a2a19d33f574b0da93 P c9a9fd5ce6b04fd5b5d2480a800f403c: Log is configured to *not* fsync() on all Append() calls
I20260810 03:29:49.647396 20467 log.cc:824] T 5e8b86f0562049a2a19d33f574b0da93 P f01667cb33c543839000d6eb9fb758d4: Log is configured to *not* fsync() on all Append() calls
I20260810 03:29:49.650408 20467 tablet_bootstrap.cc:492] T 5e8b86f0562049a2a19d33f574b0da93 P f01667cb33c543839000d6eb9fb758d4: No bootstrap required, opened a new log
I20260810 03:29:49.650395 20466 tablet_bootstrap.cc:654] T 5e8b86f0562049a2a19d33f574b0da93 P fac5507619954c158ebdd47a7d73fdc6: Neither blocks nor log segments found. Creating new log.
I20260810 03:29:49.650676 20467 ts_tablet_manager.cc:1397] T 5e8b86f0562049a2a19d33f574b0da93 P f01667cb33c543839000d6eb9fb758d4: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260810 03:29:49.650780 20465 tablet_bootstrap.cc:492] T 5e8b86f0562049a2a19d33f574b0da93 P c9a9fd5ce6b04fd5b5d2480a800f403c: No bootstrap required, opened a new log
I20260810 03:29:49.651062 20465 ts_tablet_manager.cc:1397] T 5e8b86f0562049a2a19d33f574b0da93 P c9a9fd5ce6b04fd5b5d2480a800f403c: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260810 03:29:49.651825 20466 log.cc:824] T 5e8b86f0562049a2a19d33f574b0da93 P fac5507619954c158ebdd47a7d73fdc6: Log is configured to *not* fsync() on all Append() calls
I20260810 03:29:49.655108 20466 tablet_bootstrap.cc:492] T 5e8b86f0562049a2a19d33f574b0da93 P fac5507619954c158ebdd47a7d73fdc6: No bootstrap required, opened a new log
I20260810 03:29:49.655385 20466 ts_tablet_manager.cc:1397] T 5e8b86f0562049a2a19d33f574b0da93 P fac5507619954c158ebdd47a7d73fdc6: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260810 03:29:49.658726 20467 raft_consensus.cc:348] T 5e8b86f0562049a2a19d33f574b0da93 P f01667cb33c543839000d6eb9fb758d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9a9fd5ce6b04fd5b5d2480a800f403c" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 44627 } } peers { permanent_uuid: "f01667cb33c543839000d6eb9fb758d4" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 36973 } } peers { permanent_uuid: "fac5507619954c158ebdd47a7d73fdc6" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 38339 } }
I20260810 03:29:49.659091 20467 raft_consensus.cc:374] T 5e8b86f0562049a2a19d33f574b0da93 P f01667cb33c543839000d6eb9fb758d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:29:49.659214 20467 raft_consensus.cc:729] T 5e8b86f0562049a2a19d33f574b0da93 P f01667cb33c543839000d6eb9fb758d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f01667cb33c543839000d6eb9fb758d4, State: Initialized, Role: FOLLOWER
I20260810 03:29:49.659795 20467 consensus_queue.cc:260] T 5e8b86f0562049a2a19d33f574b0da93 P f01667cb33c543839000d6eb9fb758d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9a9fd5ce6b04fd5b5d2480a800f403c" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 44627 } } peers { permanent_uuid: "f01667cb33c543839000d6eb9fb758d4" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 36973 } } peers { permanent_uuid: "fac5507619954c158ebdd47a7d73fdc6" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 38339 } }
I20260810 03:29:49.660831 20230 heartbeater.cc:503] Master 127.27.161.62:34901 was elected leader, sending a full tablet report...
I20260810 03:29:49.661525 20467 ts_tablet_manager.cc:1428] T 5e8b86f0562049a2a19d33f574b0da93 P f01667cb33c543839000d6eb9fb758d4: Time spent starting tablet: real 0.011s user 0.010s sys 0.000s
I20260810 03:29:49.661367 20465 raft_consensus.cc:348] T 5e8b86f0562049a2a19d33f574b0da93 P c9a9fd5ce6b04fd5b5d2480a800f403c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9a9fd5ce6b04fd5b5d2480a800f403c" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 44627 } } peers { permanent_uuid: "f01667cb33c543839000d6eb9fb758d4" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 36973 } } peers { permanent_uuid: "fac5507619954c158ebdd47a7d73fdc6" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 38339 } }
I20260810 03:29:49.662003 20465 raft_consensus.cc:374] T 5e8b86f0562049a2a19d33f574b0da93 P c9a9fd5ce6b04fd5b5d2480a800f403c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:29:49.662168 20465 raft_consensus.cc:729] T 5e8b86f0562049a2a19d33f574b0da93 P c9a9fd5ce6b04fd5b5d2480a800f403c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9a9fd5ce6b04fd5b5d2480a800f403c, State: Initialized, Role: FOLLOWER
I20260810 03:29:49.662992 20465 consensus_queue.cc:260] T 5e8b86f0562049a2a19d33f574b0da93 P c9a9fd5ce6b04fd5b5d2480a800f403c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9a9fd5ce6b04fd5b5d2480a800f403c" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 44627 } } peers { permanent_uuid: "f01667cb33c543839000d6eb9fb758d4" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 36973 } } peers { permanent_uuid: "fac5507619954c158ebdd47a7d73fdc6" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 38339 } }
I20260810 03:29:49.664175 20438 heartbeater.cc:503] Master 127.27.161.62:34901 was elected leader, sending a full tablet report...
I20260810 03:29:49.665655 20466 raft_consensus.cc:348] T 5e8b86f0562049a2a19d33f574b0da93 P fac5507619954c158ebdd47a7d73fdc6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9a9fd5ce6b04fd5b5d2480a800f403c" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 44627 } } peers { permanent_uuid: "f01667cb33c543839000d6eb9fb758d4" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 36973 } } peers { permanent_uuid: "fac5507619954c158ebdd47a7d73fdc6" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 38339 } }
I20260810 03:29:49.665987 20465 ts_tablet_manager.cc:1428] T 5e8b86f0562049a2a19d33f574b0da93 P c9a9fd5ce6b04fd5b5d2480a800f403c: Time spent starting tablet: real 0.015s user 0.013s sys 0.000s
I20260810 03:29:49.666121 20466 raft_consensus.cc:374] T 5e8b86f0562049a2a19d33f574b0da93 P fac5507619954c158ebdd47a7d73fdc6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:29:49.666291 20466 raft_consensus.cc:729] T 5e8b86f0562049a2a19d33f574b0da93 P fac5507619954c158ebdd47a7d73fdc6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fac5507619954c158ebdd47a7d73fdc6, State: Initialized, Role: FOLLOWER
I20260810 03:29:49.666873 20466 consensus_queue.cc:260] T 5e8b86f0562049a2a19d33f574b0da93 P fac5507619954c158ebdd47a7d73fdc6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9a9fd5ce6b04fd5b5d2480a800f403c" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 44627 } } peers { permanent_uuid: "f01667cb33c543839000d6eb9fb758d4" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 36973 } } peers { permanent_uuid: "fac5507619954c158ebdd47a7d73fdc6" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 38339 } }
I20260810 03:29:49.670961 20466 ts_tablet_manager.cc:1428] T 5e8b86f0562049a2a19d33f574b0da93 P fac5507619954c158ebdd47a7d73fdc6: Time spent starting tablet: real 0.015s user 0.016s sys 0.000s
W20260810 03:29:49.713395 20231 tablet.cc:2369] T 5e8b86f0562049a2a19d33f574b0da93 P f01667cb33c543839000d6eb9fb758d4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 03:29:49.771108 20440 tablet.cc:2369] T 5e8b86f0562049a2a19d33f574b0da93 P c9a9fd5ce6b04fd5b5d2480a800f403c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 03:29:49.844594 20477 raft_consensus.cc:482] T 5e8b86f0562049a2a19d33f574b0da93 P fac5507619954c158ebdd47a7d73fdc6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 03:29:49.844885 20477 raft_consensus.cc:504] T 5e8b86f0562049a2a19d33f574b0da93 P fac5507619954c158ebdd47a7d73fdc6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9a9fd5ce6b04fd5b5d2480a800f403c" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 44627 } } peers { permanent_uuid: "f01667cb33c543839000d6eb9fb758d4" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 36973 } } peers { permanent_uuid: "fac5507619954c158ebdd47a7d73fdc6" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 38339 } }
I20260810 03:29:49.846616 20477 leader_election.cc:288] T 5e8b86f0562049a2a19d33f574b0da93 P fac5507619954c158ebdd47a7d73fdc6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c9a9fd5ce6b04fd5b5d2480a800f403c (127.27.161.3:44627), f01667cb33c543839000d6eb9fb758d4 (127.27.161.2:36973)
I20260810 03:29:49.854461 20460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:29:49.847097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.161.1:38709 (local address 127.27.161.3:44627)
0810 03:29:49.847302 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0810 03:29:49.847308 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:29:49.847856 (+ 548us) server_negotiation.cc:407] Connection header received
0810 03:29:49.848057 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:29:49.848061 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:29:49.848155 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:29:49.848234 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:29:49.848995 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:29:49.849904 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:29:49.852661 (+ 2757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:29:49.853436 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:29:49.853560 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:29:49.853619 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0810 03:29:49.854090 (+ 471us) server_negotiation.cc:299] Negotiation successful
0810 03:29:49.854239 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260810 03:29:49.854637 20480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:29:49.846862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.3:44627 (local address 127.27.161.1:38709)
0810 03:29:49.847739 (+ 877us) negotiation.cc:107] Waiting for socket to connect
0810 03:29:49.847754 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 03:29:49.847905 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:29:49.848410 (+ 505us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:29:49.848416 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:29:49.848445 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:29:49.848807 (+ 362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:29:49.848821 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:29:49.850096 (+ 1275us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:29:49.850103 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:29:49.852501 (+ 2398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:29:49.852519 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:29:49.853653 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:29:49.853661 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:29:49.853899 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:29:49.853941 (+ 42us) client_negotiation.cc:769] Sending connection context
0810 03:29:49.854059 (+ 118us) client_negotiation.cc:240] Negotiation successful
0810 03:29:49.854221 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":763,"threads_started":1}
I20260810 03:29:49.858824 20369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e8b86f0562049a2a19d33f574b0da93" candidate_uuid: "fac5507619954c158ebdd47a7d73fdc6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9a9fd5ce6b04fd5b5d2480a800f403c" is_pre_election: true
I20260810 03:29:49.859325 20369 raft_consensus.cc:2454] T 5e8b86f0562049a2a19d33f574b0da93 P c9a9fd5ce6b04fd5b5d2480a800f403c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fac5507619954c158ebdd47a7d73fdc6 in term 0.
I20260810 03:29:49.860204 20459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:29:49.847097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.161.1:42217 (local address 127.27.161.2:36973)
0810 03:29:49.847302 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0810 03:29:49.847308 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:29:49.854972 (+ 7664us) server_negotiation.cc:407] Connection header received
0810 03:29:49.855138 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:29:49.855142 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:29:49.855207 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:29:49.855277 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:29:49.855709 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:29:49.856485 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:29:49.858530 (+ 2045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:29:49.859383 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:29:49.859516 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:29:49.859597 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0810 03:29:49.859913 (+ 316us) server_negotiation.cc:299] Negotiation successful
0810 03:29:49.860013 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260810 03:29:49.860064 19867 leader_election.cc:302] T 5e8b86f0562049a2a19d33f574b0da93 P fac5507619954c158ebdd47a7d73fdc6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c9a9fd5ce6b04fd5b5d2480a800f403c, fac5507619954c158ebdd47a7d73fdc6; no voters:
I20260810 03:29:49.860363 20480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:29:49.846955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.2:36973 (local address 127.27.161.1:42217)
0810 03:29:49.854875 (+ 7920us) negotiation.cc:107] Waiting for socket to connect
0810 03:29:49.854896 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0810 03:29:49.855027 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:29:49.855305 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:29:49.855311 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:29:49.855336 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:29:49.855511 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:29:49.855521 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:29:49.856612 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:29:49.856618 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:29:49.858393 (+ 1775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:29:49.858408 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:29:49.859540 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:29:49.859546 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:29:49.859739 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:29:49.859768 (+ 29us) client_negotiation.cc:769] Sending connection context
0810 03:29:49.859880 (+ 112us) client_negotiation.cc:240] Negotiation successful
0810 03:29:49.860018 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7799,"thread_start_us":495,"threads_started":1}
I20260810 03:29:49.860594 20477 raft_consensus.cc:2781] T 5e8b86f0562049a2a19d33f574b0da93 P fac5507619954c158ebdd47a7d73fdc6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 03:29:49.860762 20477 raft_consensus.cc:482] T 5e8b86f0562049a2a19d33f574b0da93 P fac5507619954c158ebdd47a7d73fdc6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 03:29:49.860863 20477 raft_consensus.cc:3037] T 5e8b86f0562049a2a19d33f574b0da93 P fac5507619954c158ebdd47a7d73fdc6 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:29:49.861377 20164 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e8b86f0562049a2a19d33f574b0da93" candidate_uuid: "fac5507619954c158ebdd47a7d73fdc6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f01667cb33c543839000d6eb9fb758d4" is_pre_election: true
I20260810 03:29:49.861824 20164 raft_consensus.cc:2454] T 5e8b86f0562049a2a19d33f574b0da93 P f01667cb33c543839000d6eb9fb758d4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fac5507619954c158ebdd47a7d73fdc6 in term 0.
I20260810 03:29:49.863386 20477 raft_consensus.cc:504] T 5e8b86f0562049a2a19d33f574b0da93 P fac5507619954c158ebdd47a7d73fdc6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9a9fd5ce6b04fd5b5d2480a800f403c" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 44627 } } peers { permanent_uuid: "f01667cb33c543839000d6eb9fb758d4" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 36973 } } peers { permanent_uuid: "fac5507619954c158ebdd47a7d73fdc6" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 38339 } }
I20260810 03:29:49.863999 20477 leader_election.cc:288] T 5e8b86f0562049a2a19d33f574b0da93 P fac5507619954c158ebdd47a7d73fdc6 [CANDIDATE]: Term 1 election: Requested vote from peers c9a9fd5ce6b04fd5b5d2480a800f403c (127.27.161.3:44627), f01667cb33c543839000d6eb9fb758d4 (127.27.161.2:36973)
I20260810 03:29:49.864413 20369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e8b86f0562049a2a19d33f574b0da93" candidate_uuid: "fac5507619954c158ebdd47a7d73fdc6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9a9fd5ce6b04fd5b5d2480a800f403c"
I20260810 03:29:49.864419 20164 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e8b86f0562049a2a19d33f574b0da93" candidate_uuid: "fac5507619954c158ebdd47a7d73fdc6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f01667cb33c543839000d6eb9fb758d4"
I20260810 03:29:49.864589 20164 raft_consensus.cc:3037] T 5e8b86f0562049a2a19d33f574b0da93 P f01667cb33c543839000d6eb9fb758d4 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:29:49.864604 20369 raft_consensus.cc:3037] T 5e8b86f0562049a2a19d33f574b0da93 P c9a9fd5ce6b04fd5b5d2480a800f403c [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:29:49.866766 20369 raft_consensus.cc:2454] T 5e8b86f0562049a2a19d33f574b0da93 P c9a9fd5ce6b04fd5b5d2480a800f403c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fac5507619954c158ebdd47a7d73fdc6 in term 1.
I20260810 03:29:49.866768 20164 raft_consensus.cc:2454] T 5e8b86f0562049a2a19d33f574b0da93 P f01667cb33c543839000d6eb9fb758d4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fac5507619954c158ebdd47a7d73fdc6 in term 1.
I20260810 03:29:49.867162 19869 leader_election.cc:302] T 5e8b86f0562049a2a19d33f574b0da93 P fac5507619954c158ebdd47a7d73fdc6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f01667cb33c543839000d6eb9fb758d4, fac5507619954c158ebdd47a7d73fdc6; no voters:
I20260810 03:29:49.867458 20477 raft_consensus.cc:2781] T 5e8b86f0562049a2a19d33f574b0da93 P fac5507619954c158ebdd47a7d73fdc6 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:29:49.867897 20477 raft_consensus.cc:686] T 5e8b86f0562049a2a19d33f574b0da93 P fac5507619954c158ebdd47a7d73fdc6 [term 1 LEADER]: Becoming Leader. State: Replica: fac5507619954c158ebdd47a7d73fdc6, State: Running, Role: LEADER
I20260810 03:29:49.868392 20477 consensus_queue.cc:237] T 5e8b86f0562049a2a19d33f574b0da93 P fac5507619954c158ebdd47a7d73fdc6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9a9fd5ce6b04fd5b5d2480a800f403c" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 44627 } } peers { permanent_uuid: "f01667cb33c543839000d6eb9fb758d4" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 36973 } } peers { permanent_uuid: "fac5507619954c158ebdd47a7d73fdc6" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 38339 } }
I20260810 03:29:49.874640 19734 catalog_manager.cc:5495] T 5e8b86f0562049a2a19d33f574b0da93 P fac5507619954c158ebdd47a7d73fdc6 reported cstate change: term changed from 0 to 1, leader changed from <none> to fac5507619954c158ebdd47a7d73fdc6 (127.27.161.1). New cstate: current_term: 1 leader_uuid: "fac5507619954c158ebdd47a7d73fdc6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9a9fd5ce6b04fd5b5d2480a800f403c" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 44627 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f01667cb33c543839000d6eb9fb758d4" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 36973 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fac5507619954c158ebdd47a7d73fdc6" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 38339 } health_report { overall_health: HEALTHY } } }
W20260810 03:29:49.887576 20032 tablet.cc:2369] T 5e8b86f0562049a2a19d33f574b0da93 P fac5507619954c158ebdd47a7d73fdc6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 03:29:49.918509 20463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:29:49.913555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39660 (local address 127.27.161.1:38339)
0810 03:29:49.913741 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0810 03:29:49.913746 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:29:49.913792 (+ 46us) server_negotiation.cc:407] Connection header received
0810 03:29:49.914024 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:29:49.914028 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:29:49.914112 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:29:49.914239 (+ 127us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 03:29:49.914554 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:29:49.915274 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:29:49.916620 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:29:49.917025 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:29:49.917116 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:29:49.917526 (+ 410us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 03:29:49.918073 (+ 547us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 03:29:49.918190 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0810 03:29:49.918237 (+ 47us) server_negotiation.cc:299] Negotiation successful
0810 03:29:49.918302 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260810 03:29:49.926312 20369 raft_consensus.cc:1261] T 5e8b86f0562049a2a19d33f574b0da93 P c9a9fd5ce6b04fd5b5d2480a800f403c [term 1 FOLLOWER]: Refusing update from remote peer fac5507619954c158ebdd47a7d73fdc6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:29:49.926311 20164 raft_consensus.cc:1261] T 5e8b86f0562049a2a19d33f574b0da93 P f01667cb33c543839000d6eb9fb758d4 [term 1 FOLLOWER]: Refusing update from remote peer fac5507619954c158ebdd47a7d73fdc6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:29:49.927255 20477 consensus_queue.cc:1035] T 5e8b86f0562049a2a19d33f574b0da93 P fac5507619954c158ebdd47a7d73fdc6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f01667cb33c543839000d6eb9fb758d4" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 36973 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:29:49.927533 20481 consensus_queue.cc:1035] T 5e8b86f0562049a2a19d33f574b0da93 P fac5507619954c158ebdd47a7d73fdc6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9a9fd5ce6b04fd5b5d2480a800f403c" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 44627 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:29:49.939023 20484 mvcc.cc:205] Tried to move back new op lower bound from 7316818288328785920 to 7316818288104747008. Current Snapshot: MvccSnapshot[applied={T|T < 7316818288328785920}]
I20260810 03:29:49.941896 20487 mvcc.cc:205] Tried to move back new op lower bound from 7316818288328785920 to 7316818288104747008. Current Snapshot: MvccSnapshot[applied={T|T < 7316818288328785920}]
I20260810 03:29:49.956609 19734 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53818:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 03:29:49.956918 19734 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53818:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 03:29:49.961259 19734 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b85f2c98e9de4e5bbb5359e23783aaa0: Sending DeleteTablet for 3 replicas of tablet 5e8b86f0562049a2a19d33f574b0da93
I20260810 03:29:49.962457 20340 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e8b86f0562049a2a19d33f574b0da93 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:29:49 UTC) from {username='kudu', principal='kudu/127.27.161.62@KRBTEST.COM'} at 127.0.0.1:53128
I20260810 03:29:49.962831 20139 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e8b86f0562049a2a19d33f574b0da93 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:29:49 UTC) from {username='kudu', principal='kudu/127.27.161.62@KRBTEST.COM'} at 127.0.0.1:34098
I20260810 03:29:49.963428 20499 tablet_replica.cc:330] T 5e8b86f0562049a2a19d33f574b0da93 P c9a9fd5ce6b04fd5b5d2480a800f403c: stopping tablet replica
I20260810 03:29:49.963871 20499 raft_consensus.cc:2229] T 5e8b86f0562049a2a19d33f574b0da93 P c9a9fd5ce6b04fd5b5d2480a800f403c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 03:29:49.963909 19936 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e8b86f0562049a2a19d33f574b0da93 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:29:49 UTC) from {username='kudu', principal='kudu/127.27.161.62@KRBTEST.COM'} at 127.0.0.1:39654
I20260810 03:29:49.964440 20499 raft_consensus.cc:2258] T 5e8b86f0562049a2a19d33f574b0da93 P c9a9fd5ce6b04fd5b5d2480a800f403c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:29:49.966377 20499 ts_tablet_manager.cc:1905] T 5e8b86f0562049a2a19d33f574b0da93 P c9a9fd5ce6b04fd5b5d2480a800f403c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:29:49.973363 20499 ts_tablet_manager.cc:1918] T 5e8b86f0562049a2a19d33f574b0da93 P c9a9fd5ce6b04fd5b5d2480a800f403c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 03:29:49.973579 20499 log.cc:1196] T 5e8b86f0562049a2a19d33f574b0da93 P c9a9fd5ce6b04fd5b5d2480a800f403c: Deleting WAL directory at /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/minicluster-data/ts-2/wal/wals/5e8b86f0562049a2a19d33f574b0da93
I20260810 03:29:49.974153 20499 ts_tablet_manager.cc:1939] T 5e8b86f0562049a2a19d33f574b0da93 P c9a9fd5ce6b04fd5b5d2480a800f403c: Deleting consensus metadata
I20260810 03:29:49.974339 28292 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iBH5/build/asan/bin/kudu pid 19842
I20260810 03:29:49.974511 20504 tablet_replica.cc:330] T 5e8b86f0562049a2a19d33f574b0da93 P fac5507619954c158ebdd47a7d73fdc6: stopping tablet replica
I20260810 03:29:49.974987 20504 raft_consensus.cc:2229] T 5e8b86f0562049a2a19d33f574b0da93 P fac5507619954c158ebdd47a7d73fdc6 [term 1 LEADER]: Raft consensus shutting down.
I20260810 03:29:49.975796 20504 raft_consensus.cc:2258] T 5e8b86f0562049a2a19d33f574b0da93 P fac5507619954c158ebdd47a7d73fdc6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:29:49.976109 19711 catalog_manager.cc:4841] TS c9a9fd5ce6b04fd5b5d2480a800f403c (127.27.161.3:44627): tablet 5e8b86f0562049a2a19d33f574b0da93 (table test-table [id=a88440a4cc834973a1247345a76b6fdc]) successfully deleted
I20260810 03:29:49.976372 20500 tablet_replica.cc:330] T 5e8b86f0562049a2a19d33f574b0da93 P f01667cb33c543839000d6eb9fb758d4: stopping tablet replica
I20260810 03:29:49.976747 20500 raft_consensus.cc:2229] T 5e8b86f0562049a2a19d33f574b0da93 P f01667cb33c543839000d6eb9fb758d4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 03:29:49.977164 20500 raft_consensus.cc:2258] T 5e8b86f0562049a2a19d33f574b0da93 P f01667cb33c543839000d6eb9fb758d4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:29:49.979290 20504 ts_tablet_manager.cc:1905] T 5e8b86f0562049a2a19d33f574b0da93 P fac5507619954c158ebdd47a7d73fdc6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:29:49.982188 20500 ts_tablet_manager.cc:1905] T 5e8b86f0562049a2a19d33f574b0da93 P f01667cb33c543839000d6eb9fb758d4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:29:49.986951 20504 ts_tablet_manager.cc:1918] T 5e8b86f0562049a2a19d33f574b0da93 P fac5507619954c158ebdd47a7d73fdc6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 03:29:49.987238 20504 log.cc:1196] T 5e8b86f0562049a2a19d33f574b0da93 P fac5507619954c158ebdd47a7d73fdc6: Deleting WAL directory at /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/minicluster-data/ts-0/wal/wals/5e8b86f0562049a2a19d33f574b0da93
I20260810 03:29:49.987850 20504 ts_tablet_manager.cc:1939] T 5e8b86f0562049a2a19d33f574b0da93 P fac5507619954c158ebdd47a7d73fdc6: Deleting consensus metadata
I20260810 03:29:49.989032 20500 ts_tablet_manager.cc:1918] T 5e8b86f0562049a2a19d33f574b0da93 P f01667cb33c543839000d6eb9fb758d4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 03:29:49.989966 19712 catalog_manager.cc:4841] TS fac5507619954c158ebdd47a7d73fdc6 (127.27.161.1:38339): tablet 5e8b86f0562049a2a19d33f574b0da93 (table test-table [id=a88440a4cc834973a1247345a76b6fdc]) successfully deleted
I20260810 03:29:49.994510 20500 log.cc:1196] T 5e8b86f0562049a2a19d33f574b0da93 P f01667cb33c543839000d6eb9fb758d4: Deleting WAL directory at /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/minicluster-data/ts-1/wal/wals/5e8b86f0562049a2a19d33f574b0da93
I20260810 03:29:49.994963 20500 ts_tablet_manager.cc:1939] T 5e8b86f0562049a2a19d33f574b0da93 P f01667cb33c543839000d6eb9fb758d4: Deleting consensus metadata
I20260810 03:29:49.996518 19713 catalog_manager.cc:4841] TS f01667cb33c543839000d6eb9fb758d4 (127.27.161.2:36973): tablet 5e8b86f0562049a2a19d33f574b0da93 (table test-table [id=a88440a4cc834973a1247345a76b6fdc]) successfully deleted
Aug 10 03:29:49 dist-test-slave-vpvm krb5kdc[19667](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332587, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.161.1@KRBTEST.COM
I20260810 03:29:50.006225 20463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:29:49.975548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39674 (local address 127.27.161.1:38339)
0810 03:29:49.975752 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0810 03:29:49.975759 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:29:49.985000 (+ 9241us) server_negotiation.cc:407] Connection header received
0810 03:29:49.985556 (+ 556us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:29:49.985563 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:29:49.985701 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:29:49.985789 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:29:49.988018 (+ 2229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:29:49.989238 (+ 1220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:29:49.991999 (+ 2761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:29:49.992711 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:29:49.992848 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:29:50.000011 (+ 7163us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:29:50.000024 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:29:50.000041 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:29:50.000102 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:29:50.003250 (+ 3148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:29:50.004041 (+ 791us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:29:50.004049 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:29:50.004052 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:29:50.004217 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:29:50.004605 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:29:50.004609 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:29:50.004611 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:29:50.004961 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:29:50.005289 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0810 03:29:50.005600 (+ 311us) server_negotiation.cc:299] Negotiation successful
0810 03:29:50.005844 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260810 03:29:50.006930 20024 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:29:50.127483 28292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iBH5/build/asan/bin/kudu with pid 19842
I20260810 03:29:50.165206 28292 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iBH5/build/asan/bin/kudu pid 20038
Aug 10 03:29:50 dist-test-slave-vpvm krb5kdc[19667](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332587, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.161.2@KRBTEST.COM
I20260810 03:29:50.184811 20459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:29:50.166012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34108 (local address 127.27.161.2:36973)
0810 03:29:50.166213 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0810 03:29:50.166219 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:29:50.166241 (+ 22us) server_negotiation.cc:407] Connection header received
0810 03:29:50.166682 (+ 441us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:29:50.166689 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:29:50.166783 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:29:50.166898 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:29:50.168252 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:29:50.169375 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:29:50.170914 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:29:50.171623 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:29:50.171773 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:29:50.176670 (+ 4897us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:29:50.176702 (+ 32us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:29:50.176729 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:29:50.176796 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:29:50.181274 (+ 4478us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:29:50.182239 (+ 965us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:29:50.182251 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:29:50.182255 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:29:50.182458 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:29:50.182865 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:29:50.182871 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:29:50.182875 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:29:50.183538 (+ 663us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:29:50.183901 (+ 363us) server_negotiation.cc:1035] Waiting for connection context
0810 03:29:50.184091 (+ 190us) server_negotiation.cc:299] Negotiation successful
0810 03:29:50.184346 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260810 03:29:50.185552 20221 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:29:50.280088 28292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iBH5/build/asan/bin/kudu with pid 20038
I20260810 03:29:50.306425 28292 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iBH5/build/asan/bin/kudu pid 20239
Aug 10 03:29:50 dist-test-slave-vpvm krb5kdc[19667](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332587, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.161.3@KRBTEST.COM
I20260810 03:29:50.322440 20460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:29:50.307192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53134 (local address 127.27.161.3:44627)
0810 03:29:50.307371 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0810 03:29:50.307378 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:29:50.307449 (+ 71us) server_negotiation.cc:407] Connection header received
0810 03:29:50.307638 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:29:50.307643 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:29:50.307723 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:29:50.307822 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:29:50.309157 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:29:50.309877 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:29:50.311306 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:29:50.311717 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:29:50.311832 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:29:50.316457 (+ 4625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:29:50.316477 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:29:50.316496 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:29:50.316536 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:29:50.319319 (+ 2783us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:29:50.320050 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:29:50.320058 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:29:50.320061 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:29:50.320224 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:29:50.320608 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:29:50.320612 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:29:50.320614 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:29:50.321023 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:29:50.321326 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0810 03:29:50.321654 (+ 328us) server_negotiation.cc:299] Negotiation successful
0810 03:29:50.321897 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260810 03:29:50.323263 20430 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:29:50.411787 28292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iBH5/build/asan/bin/kudu with pid 20239
I20260810 03:29:50.435096 28292 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iBH5/build/asan/bin/kudu pid 19687
I20260810 03:29:50.436221 19786 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:29:50.547909 28292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iBH5/build/asan/bin/kudu with pid 19687
I20260810 03:30:10.579259 28292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iBH5/build/asan/bin/kudu
/tmp/dist-test-task24iBH5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.161.62:34901
--webserver_interface=127.27.161.62
--webserver_port=34355
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.62.keytab
--principal=kudu/127.27.161.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.161.20:42379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.161.62:34901
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:30:10.956250 20512 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:30:10.956585 20512 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:30:10.956738 20512 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:30:10.964093 20512 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260810 03:30:10.964210 20512 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260810 03:30:10.964269 20512 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260810 03:30:10.964341 20512 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:30:10.964416 20512 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:30:10.964514 20512 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:30:10.964587 20512 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:30:10.964663 20512 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:30:10.974747 20512 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.161.20:42379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.27.161.62:34901
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.62.keytab
--principal=kudu/127.27.161.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.161.62:34901
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.161.62
--webserver_port=34355
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:30:10.976107 20512 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:30:10.977772 20512 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:30:11.003448 20524 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:30:11.003568 20523 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:30:11.003479 20527 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:30:11.004268 20512 server_base.cc:1034] running on GCE node
I20260810 03:30:11.005242 20512 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:30:11.006942 20512 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:30:11.008185 20512 hybrid_clock.cc:648] HybridClock initialized: now 1786332611008111 us; error 55 us; skew 500 ppm
I20260810 03:30:11.008598 20512 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:30:11 dist-test-slave-vpvm krb5kdc[19667](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332611, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.161.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:30:11.017246 20512 init.cc:377] Logged in from keytab as kudu/127.27.161.62@KRBTEST.COM (short username kudu)
I20260810 03:30:11.019162 20512 webserver.cc:460] Webserver started at http://127.27.161.62:34355/ using document root <none> and password file <none>
I20260810 03:30:11.020021 20512 fs_manager.cc:359] Metadata directory not provided
I20260810 03:30:11.020154 20512 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:30:11.026794 20512 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260810 03:30:11.029923 20538 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:30:11.031505 20512 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260810 03:30:11.031706 20512 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/minicluster-data/master-0/data,/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/minicluster-data/master-0/wal
uuid: "b85f2c98e9de4e5bbb5359e23783aaa0"
format_stamp: "Formatted at 2026-08-10 03:29:47 on dist-test-slave-vpvm"
server_key: "1109bcb983d5b2e9d420ac696de1a21c"
server_key_iv: "d46666874395a5ac58653f3aa33db5cb"
server_key_version: "encryptionkey@0"
I20260810 03:30:11.032826 20512 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:30:11.061417 20512 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:30:11.066726 20512 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:30:11.067061 20512 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:30:11.087543 20512 rpc_server.cc:307] RPC server started. Bound to: 127.27.161.62:34901
I20260810 03:30:11.087589 20619 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.161.62:34901 every 8 connection(s)
I20260810 03:30:11.089239 20512 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/minicluster-data/master-0/data/info.pb
I20260810 03:30:11.096967 28292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iBH5/build/asan/bin/kudu as pid 20512
I20260810 03:30:11.097159 20621 sys_catalog.cc:263] Verifying existing consensus state
I20260810 03:30:11.098090 28292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iBH5/build/asan/bin/kudu
/tmp/dist-test-task24iBH5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.161.1:38339
--local_ip_for_outbound_sockets=127.27.161.1
--tserver_master_addrs=127.27.161.62:34901
--webserver_port=43831
--webserver_interface=127.27.161.1
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.1.keytab
--principal=kudu/127.27.161.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.161.20:42379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 03:30:11.102241 20621 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b85f2c98e9de4e5bbb5359e23783aaa0: Bootstrap starting.
I20260810 03:30:11.125497 20621 log.cc:824] T 00000000000000000000000000000000 P b85f2c98e9de4e5bbb5359e23783aaa0: Log is configured to *not* fsync() on all Append() calls
I20260810 03:30:11.139504 20621 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b85f2c98e9de4e5bbb5359e23783aaa0: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 03:30:11.140061 20621 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b85f2c98e9de4e5bbb5359e23783aaa0: Bootstrap complete.
I20260810 03:30:11.149544 20621 raft_consensus.cc:348] T 00000000000000000000000000000000 P b85f2c98e9de4e5bbb5359e23783aaa0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b85f2c98e9de4e5bbb5359e23783aaa0" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 34901 } }
I20260810 03:30:11.150511 20621 raft_consensus.cc:729] T 00000000000000000000000000000000 P b85f2c98e9de4e5bbb5359e23783aaa0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b85f2c98e9de4e5bbb5359e23783aaa0, State: Initialized, Role: FOLLOWER
I20260810 03:30:11.151166 20621 consensus_queue.cc:260] T 00000000000000000000000000000000 P b85f2c98e9de4e5bbb5359e23783aaa0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b85f2c98e9de4e5bbb5359e23783aaa0" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 34901 } }
I20260810 03:30:11.151400 20621 raft_consensus.cc:388] T 00000000000000000000000000000000 P b85f2c98e9de4e5bbb5359e23783aaa0 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 03:30:11.151526 20621 raft_consensus.cc:482] T 00000000000000000000000000000000 P b85f2c98e9de4e5bbb5359e23783aaa0 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 03:30:11.151700 20621 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b85f2c98e9de4e5bbb5359e23783aaa0 [term 1 FOLLOWER]: Advancing to term 2
I20260810 03:30:11.155941 20621 raft_consensus.cc:504] T 00000000000000000000000000000000 P b85f2c98e9de4e5bbb5359e23783aaa0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b85f2c98e9de4e5bbb5359e23783aaa0" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 34901 } }
I20260810 03:30:11.156500 20621 leader_election.cc:302] T 00000000000000000000000000000000 P b85f2c98e9de4e5bbb5359e23783aaa0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b85f2c98e9de4e5bbb5359e23783aaa0; no voters:
I20260810 03:30:11.157157 20621 leader_election.cc:288] T 00000000000000000000000000000000 P b85f2c98e9de4e5bbb5359e23783aaa0 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260810 03:30:11.157311 20627 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b85f2c98e9de4e5bbb5359e23783aaa0 [term 2 FOLLOWER]: Leader election won for term 2
I20260810 03:30:11.158521 20627 raft_consensus.cc:686] T 00000000000000000000000000000000 P b85f2c98e9de4e5bbb5359e23783aaa0 [term 2 LEADER]: Becoming Leader. State: Replica: b85f2c98e9de4e5bbb5359e23783aaa0, State: Running, Role: LEADER
I20260810 03:30:11.159163 20627 consensus_queue.cc:237] T 00000000000000000000000000000000 P b85f2c98e9de4e5bbb5359e23783aaa0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b85f2c98e9de4e5bbb5359e23783aaa0" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 34901 } }
I20260810 03:30:11.159554 20621 sys_catalog.cc:564] T 00000000000000000000000000000000 P b85f2c98e9de4e5bbb5359e23783aaa0 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:30:11.161907 20628 sys_catalog.cc:455] T 00000000000000000000000000000000 P b85f2c98e9de4e5bbb5359e23783aaa0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "b85f2c98e9de4e5bbb5359e23783aaa0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b85f2c98e9de4e5bbb5359e23783aaa0" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 34901 } } }
I20260810 03:30:11.162374 20628 sys_catalog.cc:458] T 00000000000000000000000000000000 P b85f2c98e9de4e5bbb5359e23783aaa0 [sys.catalog]: This master's current role is: LEADER
I20260810 03:30:11.164618 20630 sys_catalog.cc:455] T 00000000000000000000000000000000 P b85f2c98e9de4e5bbb5359e23783aaa0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b85f2c98e9de4e5bbb5359e23783aaa0. Latest consensus state: current_term: 2 leader_uuid: "b85f2c98e9de4e5bbb5359e23783aaa0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b85f2c98e9de4e5bbb5359e23783aaa0" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 34901 } } }
I20260810 03:30:11.165148 20630 sys_catalog.cc:458] T 00000000000000000000000000000000 P b85f2c98e9de4e5bbb5359e23783aaa0 [sys.catalog]: This master's current role is: LEADER
I20260810 03:30:11.166801 20634 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 03:30:11.176007 20634 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 03:30:11.177831 20634 catalog_manager.cc:1259] Loaded cluster ID: 49064dc1c4064843a5cd4d7e37ffd0d9
I20260810 03:30:11.178064 20634 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 03:30:11.183190 20634 catalog_manager.cc:1504] Loading token signing keys...
I20260810 03:30:11.185252 20634 catalog_manager.cc:5879] T 00000000000000000000000000000000 P b85f2c98e9de4e5bbb5359e23783aaa0: Loaded TSK: 0
I20260810 03:30:11.212458 20634 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b85f2c98e9de4e5bbb5359e23783aaa0: Generated new TSK 1
I20260810 03:30:11.212869 20634 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260810 03:30:11.511132 20624 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:30:11.511456 20624 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:30:11.511583 20624 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:30:11.519438 20624 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:30:11.519572 20624 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:30:11.519714 20624 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.161.1
I20260810 03:30:11.530025 20624 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.161.20:42379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.1.keytab
--principal=kudu/127.27.161.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.161.1:38339
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.161.1
--webserver_port=43831
--tserver_master_addrs=127.27.161.62:34901
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.161.1
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:30:11.531463 20624 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:30:11.532935 20624 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:30:11.548032 20659 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:30:11.548032 20666 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:30:11.548362 20660 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:30:11.550125 20624 server_base.cc:1034] running on GCE node
I20260810 03:30:11.551314 20624 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:30:11.552603 20624 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:30:11.553855 20624 hybrid_clock.cc:648] HybridClock initialized: now 1786332611553800 us; error 52 us; skew 500 ppm
I20260810 03:30:11.554258 20624 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:30:11 dist-test-slave-vpvm krb5kdc[19667](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332611, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.161.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:30:11.558121 20624 init.cc:377] Logged in from keytab as kudu/127.27.161.1@KRBTEST.COM (short username kudu)
I20260810 03:30:11.559921 20624 webserver.cc:460] Webserver started at http://127.27.161.1:43831/ using document root <none> and password file <none>
I20260810 03:30:11.560709 20624 fs_manager.cc:359] Metadata directory not provided
I20260810 03:30:11.560876 20624 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:30:11.566854 20624 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 03:30:11.569844 20680 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:30:11.571342 20624 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260810 03:30:11.571512 20624 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/minicluster-data/ts-0/data,/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/minicluster-data/ts-0/wal
uuid: "fac5507619954c158ebdd47a7d73fdc6"
format_stamp: "Formatted at 2026-08-10 03:29:48 on dist-test-slave-vpvm"
server_key: "a5851e1672c45620019b53ae23d94ea1"
server_key_iv: "ee5f60108cbc24f6eda769236ae9dc3e"
server_key_version: "encryptionkey@0"
I20260810 03:30:11.572532 20624 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:30:11.612023 20624 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:30:11.616459 20624 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:30:11.616842 20624 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:30:11.618044 20624 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:30:11.619540 20624 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:30:11.619637 20624 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:30:11.619726 20624 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:30:11.619808 20624 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:30:11.652416 20624 rpc_server.cc:307] RPC server started. Bound to: 127.27.161.1:38339
I20260810 03:30:11.652484 20858 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.161.1:38339 every 8 connection(s)
I20260810 03:30:11.653916 20624 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/minicluster-data/ts-0/data/info.pb
I20260810 03:30:11.655579 28292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iBH5/build/asan/bin/kudu as pid 20624
I20260810 03:30:11.656283 28292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iBH5/build/asan/bin/kudu
/tmp/dist-test-task24iBH5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.161.2:36973
--local_ip_for_outbound_sockets=127.27.161.2
--tserver_master_addrs=127.27.161.62:34901
--webserver_port=43657
--webserver_interface=127.27.161.2
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.2.keytab
--principal=kudu/127.27.161.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.161.20:42379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 03:30:11 dist-test-slave-vpvm krb5kdc[19667](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332611, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.161.1@KRBTEST.COM for kudu/127.27.161.62@KRBTEST.COM
I20260810 03:30:11.680496 20863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:30:11.656924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.161.1:43047 (local address 127.27.161.62:34901)
0810 03:30:11.659028 (+ 2104us) server_negotiation.cc:206] Beginning negotiation
0810 03:30:11.659039 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 03:30:11.659085 (+ 46us) server_negotiation.cc:407] Connection header received
0810 03:30:11.661069 (+ 1984us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:30:11.661086 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:30:11.661456 (+ 370us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:30:11.661781 (+ 325us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:30:11.663767 (+ 1986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:30:11.665049 (+ 1282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:30:11.667142 (+ 2093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:30:11.667921 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:30:11.668110 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:30:11.672610 (+ 4500us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:30:11.672621 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:30:11.672639 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:30:11.672692 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:30:11.675631 (+ 2939us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:30:11.676826 (+ 1195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:30:11.676837 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:30:11.676845 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:30:11.677067 (+ 222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:30:11.677566 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:30:11.677570 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:30:11.677572 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:30:11.678148 (+ 576us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:30:11.678295 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0810 03:30:11.679211 (+ 916us) server_negotiation.cc:299] Negotiation successful
0810 03:30:11.679535 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1907,"thread_start_us":1775,"threads_started":1}
I20260810 03:30:11.680907 20865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:30:11.657120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:34901 (local address 127.27.161.1:43047)
0810 03:30:11.658143 (+ 1023us) negotiation.cc:107] Waiting for socket to connect
0810 03:30:11.658206 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0810 03:30:11.660436 (+ 2230us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:30:11.662009 (+ 1573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:30:11.662019 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:30:11.662733 (+ 714us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:30:11.663486 (+ 753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:30:11.663521 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:30:11.665217 (+ 1696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:30:11.665222 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:30:11.666978 (+ 1756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:30:11.666995 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:30:11.668166 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:30:11.668172 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:30:11.668425 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:30:11.669164 (+ 739us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:30:11.669194 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:30:11.672187 (+ 2993us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:30:11.675859 (+ 3672us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:30:11.675870 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:30:11.675886 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:30:11.676638 (+ 752us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:30:11.677190 (+ 552us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:30:11.677195 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:30:11.677198 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:30:11.677448 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:30:11.678296 (+ 848us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:30:11.678303 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:30:11.678721 (+ 418us) client_negotiation.cc:769] Sending connection context
0810 03:30:11.679022 (+ 301us) client_negotiation.cc:240] Negotiation successful
0810 03:30:11.679635 (+ 613us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":665,"thread_start_us":351,"threads_started":1}
I20260810 03:30:11.684010 20859 heartbeater.cc:347] Connected to a master server at 127.27.161.62:34901
I20260810 03:30:11.684352 20859 heartbeater.cc:464] Registering TS with master...
I20260810 03:30:11.685314 20859 heartbeater.cc:511] Master 127.27.161.62:34901 requested a full tablet report, sending...
I20260810 03:30:11.687793 20569 ts_manager.cc:194] Registered new tserver with Master: fac5507619954c158ebdd47a7d73fdc6 (127.27.161.1:38339)
I20260810 03:30:11.689373 20569 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.161.1@KRBTEST.COM'} at 127.27.161.1:43047
W20260810 03:30:12.019526 20866 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:30:12.019846 20866 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:30:12.019971 20866 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:30:12.027926 20866 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:30:12.028062 20866 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:30:12.028218 20866 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.161.2
I20260810 03:30:12.038802 20866 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.161.20:42379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.2.keytab
--principal=kudu/127.27.161.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.161.2:36973
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.161.2
--webserver_port=43657
--tserver_master_addrs=127.27.161.62:34901
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.161.2
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:30:12.040210 20866 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:30:12.041635 20866 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:30:12.055609 20878 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:30:12.055730 20872 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:30:12.055716 20871 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:30:12.056867 20866 server_base.cc:1034] running on GCE node
I20260810 03:30:12.057598 20866 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:30:12.058732 20866 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:30:12.059967 20866 hybrid_clock.cc:648] HybridClock initialized: now 1786332612059892 us; error 72 us; skew 500 ppm
I20260810 03:30:12.060312 20866 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:30:12 dist-test-slave-vpvm krb5kdc[19667](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332612, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.161.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:30:12.063839 20866 init.cc:377] Logged in from keytab as kudu/127.27.161.2@KRBTEST.COM (short username kudu)
I20260810 03:30:12.065459 20866 webserver.cc:460] Webserver started at http://127.27.161.2:43657/ using document root <none> and password file <none>
I20260810 03:30:12.066269 20866 fs_manager.cc:359] Metadata directory not provided
I20260810 03:30:12.066416 20866 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:30:12.072139 20866 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 03:30:12.075147 20886 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:30:12.076642 20866 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 03:30:12.076843 20866 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/minicluster-data/ts-1/data,/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/minicluster-data/ts-1/wal
uuid: "f01667cb33c543839000d6eb9fb758d4"
format_stamp: "Formatted at 2026-08-10 03:29:48 on dist-test-slave-vpvm"
server_key: "28983bd267833e90f784d217c84c1e21"
server_key_iv: "442c9269eed320d80b27b8851ba899c6"
server_key_version: "encryptionkey@0"
I20260810 03:30:12.077890 20866 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:30:12.092388 20866 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:30:12.098035 20866 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:30:12.098340 20866 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:30:12.099529 20866 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:30:12.101310 20866 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:30:12.101428 20866 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:30:12.101540 20866 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:30:12.101611 20866 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:30:12.135054 20866 rpc_server.cc:307] RPC server started. Bound to: 127.27.161.2:36973
I20260810 03:30:12.135129 21056 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.161.2:36973 every 8 connection(s)
I20260810 03:30:12.136576 20866 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/minicluster-data/ts-1/data/info.pb
I20260810 03:30:12.145077 28292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iBH5/build/asan/bin/kudu as pid 20866
I20260810 03:30:12.145897 28292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iBH5/build/asan/bin/kudu
/tmp/dist-test-task24iBH5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.161.3:44627
--local_ip_for_outbound_sockets=127.27.161.3
--tserver_master_addrs=127.27.161.62:34901
--webserver_port=42135
--webserver_interface=127.27.161.3
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.3.keytab
--principal=kudu/127.27.161.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.161.20:42379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 03:30:12 dist-test-slave-vpvm krb5kdc[19667](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332612, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.161.2@KRBTEST.COM for kudu/127.27.161.62@KRBTEST.COM
I20260810 03:30:12.159017 20863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:30:12.139298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.161.2:58075 (local address 127.27.161.62:34901)
0810 03:30:12.139493 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0810 03:30:12.139498 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:30:12.140668 (+ 1170us) server_negotiation.cc:407] Connection header received
0810 03:30:12.142700 (+ 2032us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:30:12.142706 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:30:12.142831 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:30:12.142931 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:30:12.144708 (+ 1777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:30:12.145783 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:30:12.147602 (+ 1819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:30:12.148215 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:30:12.148364 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:30:12.152093 (+ 3729us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:30:12.152100 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:30:12.152104 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:30:12.152151 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:30:12.155351 (+ 3200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:30:12.156385 (+ 1034us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:30:12.156391 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:30:12.156394 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:30:12.156555 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:30:12.157050 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:30:12.157054 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:30:12.157065 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:30:12.157448 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:30:12.157605 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0810 03:30:12.158447 (+ 842us) server_negotiation.cc:299] Negotiation successful
0810 03:30:12.158704 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260810 03:30:12.160048 21060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:30:12.139769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:34901 (local address 127.27.161.2:58075)
0810 03:30:12.140491 (+ 722us) negotiation.cc:107] Waiting for socket to connect
0810 03:30:12.140540 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0810 03:30:12.142377 (+ 1837us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:30:12.143246 (+ 869us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:30:12.143257 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:30:12.143821 (+ 564us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:30:12.144454 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:30:12.144474 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:30:12.146028 (+ 1554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:30:12.146035 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:30:12.147411 (+ 1376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:30:12.147427 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:30:12.148444 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:30:12.148449 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:30:12.148671 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:30:12.149393 (+ 722us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:30:12.149418 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:30:12.151772 (+ 2354us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:30:12.155571 (+ 3799us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:30:12.155581 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:30:12.155628 (+ 47us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:30:12.156225 (+ 597us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:30:12.156691 (+ 466us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:30:12.156694 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:30:12.156697 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:30:12.156924 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:30:12.157596 (+ 672us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:30:12.157603 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:30:12.158041 (+ 438us) client_negotiation.cc:769] Sending connection context
0810 03:30:12.158309 (+ 268us) client_negotiation.cc:240] Negotiation successful
0810 03:30:12.158775 (+ 466us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":278,"threads_started":1}
I20260810 03:30:12.162004 21057 heartbeater.cc:347] Connected to a master server at 127.27.161.62:34901
I20260810 03:30:12.162331 21057 heartbeater.cc:464] Registering TS with master...
I20260810 03:30:12.163103 21057 heartbeater.cc:511] Master 127.27.161.62:34901 requested a full tablet report, sending...
I20260810 03:30:12.164671 20569 ts_manager.cc:194] Registered new tserver with Master: f01667cb33c543839000d6eb9fb758d4 (127.27.161.2:36973)
I20260810 03:30:12.165512 20569 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.161.2@KRBTEST.COM'} at 127.27.161.2:58075
W20260810 03:30:12.504501 21061 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:30:12.504807 21061 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:30:12.504936 21061 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:30:12.512449 21061 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:30:12.512581 21061 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:30:12.512744 21061 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.161.3
I20260810 03:30:12.522894 21061 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.161.20:42379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.3.keytab
--principal=kudu/127.27.161.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.161.3:44627
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.161.3
--webserver_port=42135
--tserver_master_addrs=127.27.161.62:34901
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.161.3
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:30:12.524317 21061 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:30:12.525655 21061 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:30:12.539683 21074 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:30:12.539644 21071 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:30:12.539978 21070 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:30:12.540786 21061 server_base.cc:1034] running on GCE node
I20260810 03:30:12.541402 21061 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:30:12.542420 21061 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:30:12.543670 21061 hybrid_clock.cc:648] HybridClock initialized: now 1786332612543616 us; error 33 us; skew 500 ppm
I20260810 03:30:12.544015 21061 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:30:12 dist-test-slave-vpvm krb5kdc[19667](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332612, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.161.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:30:12.547204 21061 init.cc:377] Logged in from keytab as kudu/127.27.161.3@KRBTEST.COM (short username kudu)
I20260810 03:30:12.548728 21061 webserver.cc:460] Webserver started at http://127.27.161.3:42135/ using document root <none> and password file <none>
I20260810 03:30:12.549443 21061 fs_manager.cc:359] Metadata directory not provided
I20260810 03:30:12.549592 21061 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:30:12.555334 21061 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.003s
I20260810 03:30:12.558368 21082 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:30:12.559888 21061 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260810 03:30:12.560063 21061 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/minicluster-data/ts-2/data,/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/minicluster-data/ts-2/wal
uuid: "c9a9fd5ce6b04fd5b5d2480a800f403c"
format_stamp: "Formatted at 2026-08-10 03:29:49 on dist-test-slave-vpvm"
server_key: "eed21fa6d563145c0f2de57944d4bf26"
server_key_iv: "c70ac530c2efd77a7ba72de0ce6a5193"
server_key_version: "encryptionkey@0"
I20260810 03:30:12.561185 21061 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:30:12.585690 21061 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:30:12.589985 21061 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:30:12.590299 21061 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:30:12.591389 21061 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:30:12.592864 21061 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:30:12.592963 21061 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:30:12.593044 21061 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:30:12.593101 21061 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:30:12.626143 21061 rpc_server.cc:307] RPC server started. Bound to: 127.27.161.3:44627
I20260810 03:30:12.626219 21255 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.161.3:44627 every 8 connection(s)
I20260810 03:30:12.627657 21061 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/minicluster-data/ts-2/data/info.pb
I20260810 03:30:12.631950 28292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iBH5/build/asan/bin/kudu as pid 21061
Aug 10 03:30:12 dist-test-slave-vpvm krb5kdc[19667](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332612, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.161.3@KRBTEST.COM for kudu/127.27.161.62@KRBTEST.COM
I20260810 03:30:12.650025 20863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:30:12.630357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.161.3:52601 (local address 127.27.161.62:34901)
0810 03:30:12.630599 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0810 03:30:12.630606 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:30:12.631938 (+ 1332us) server_negotiation.cc:407] Connection header received
0810 03:30:12.634677 (+ 2739us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:30:12.634684 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:30:12.634771 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:30:12.634859 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:30:12.636549 (+ 1690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:30:12.637350 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:30:12.639027 (+ 1677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:30:12.639515 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:30:12.639642 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:30:12.643483 (+ 3841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:30:12.643488 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:30:12.643491 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:30:12.643525 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:30:12.646626 (+ 3101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:30:12.647424 (+ 798us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:30:12.647430 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:30:12.647434 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:30:12.647587 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:30:12.648049 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:30:12.648053 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:30:12.648055 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:30:12.648346 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:30:12.648454 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0810 03:30:12.649402 (+ 948us) server_negotiation.cc:299] Negotiation successful
0810 03:30:12.649630 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260810 03:30:12.651077 21265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:30:12.633099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36592 (local address 127.27.161.62:34901)
0810 03:30:12.636778 (+ 3679us) server_negotiation.cc:206] Beginning negotiation
0810 03:30:12.636782 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:30:12.636798 (+ 16us) server_negotiation.cc:407] Connection header received
0810 03:30:12.636843 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:30:12.636846 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:30:12.636901 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:30:12.636995 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:30:12.638864 (+ 1869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:30:12.639671 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:30:12.641196 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:30:12.641626 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:30:12.641734 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:30:12.644822 (+ 3088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:30:12.644829 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:30:12.644835 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:30:12.644882 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:30:12.647863 (+ 2981us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:30:12.648665 (+ 802us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:30:12.648669 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:30:12.648684 (+ 15us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:30:12.648788 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:30:12.649212 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:30:12.649215 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:30:12.649217 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:30:12.649553 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:30:12.649836 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0810 03:30:12.650199 (+ 363us) server_negotiation.cc:299] Negotiation successful
0810 03:30:12.650410 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":3586,"spinlock_wait_cycles":2432,"thread_start_us":3420,"threads_started":1}
I20260810 03:30:12.651165 21262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:30:12.630915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:34901 (local address 127.27.161.3:52601)
0810 03:30:12.631752 (+ 837us) negotiation.cc:107] Waiting for socket to connect
0810 03:30:12.631800 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0810 03:30:12.634380 (+ 2580us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:30:12.635142 (+ 762us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:30:12.635150 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:30:12.635685 (+ 535us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:30:12.636358 (+ 673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:30:12.636379 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:30:12.637513 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:30:12.637517 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:30:12.638871 (+ 1354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:30:12.638885 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:30:12.639765 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:30:12.639773 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:30:12.640009 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:30:12.640721 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:30:12.640744 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:30:12.643245 (+ 2501us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:30:12.646792 (+ 3547us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:30:12.646799 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:30:12.646814 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:30:12.647272 (+ 458us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:30:12.647713 (+ 441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:30:12.647717 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:30:12.647719 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:30:12.647923 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:30:12.648485 (+ 562us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:30:12.648495 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:30:12.648958 (+ 463us) client_negotiation.cc:769] Sending connection context
0810 03:30:12.649283 (+ 325us) client_negotiation.cc:240] Negotiation successful
0810 03:30:12.649776 (+ 493us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":571,"thread_start_us":320,"threads_started":1}
I20260810 03:30:12.652657 21257 heartbeater.cc:347] Connected to a master server at 127.27.161.62:34901
I20260810 03:30:12.653023 21257 heartbeater.cc:464] Registering TS with master...
I20260810 03:30:12.653976 21257 heartbeater.cc:511] Master 127.27.161.62:34901 requested a full tablet report, sending...
I20260810 03:30:12.655462 20569 ts_manager.cc:194] Registered new tserver with Master: c9a9fd5ce6b04fd5b5d2480a800f403c (127.27.161.3:44627)
I20260810 03:30:12.656376 20569 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.161.3@KRBTEST.COM'} at 127.27.161.3:52601
I20260810 03:30:12.666719 28292 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 03:30:12.674027 21265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:30:12.667771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36596 (local address 127.27.161.62:34901)
0810 03:30:12.667979 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0810 03:30:12.667987 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 03:30:12.668452 (+ 465us) server_negotiation.cc:407] Connection header received
0810 03:30:12.668687 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:30:12.668691 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:30:12.668856 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:30:12.668999 (+ 143us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 03:30:12.669468 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:30:12.670279 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:30:12.671779 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:30:12.672272 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:30:12.672391 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:30:12.672965 (+ 574us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 03:30:12.673284 (+ 319us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0810 03:30:12.673690 (+ 406us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:36596: token expired
Metrics: {"server-negotiator.queue_time_us":57}
W20260810 03:30:12.674150 21265 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:36596: token expired
W20260810 03:30:12.674772 20445 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.27.161.62:34901)
I20260810 03:30:12.688041 21265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:30:12.675288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36606 (local address 127.27.161.62:34901)
0810 03:30:12.675462 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0810 03:30:12.675467 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:30:12.675485 (+ 18us) server_negotiation.cc:407] Connection header received
0810 03:30:12.675603 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:30:12.675607 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:30:12.675671 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:30:12.675748 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:30:12.677164 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:30:12.677909 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:30:12.679349 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:30:12.679766 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:30:12.679878 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:30:12.682771 (+ 2893us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:30:12.682776 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:30:12.682780 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:30:12.682819 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:30:12.685193 (+ 2374us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:30:12.685920 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:30:12.685926 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:30:12.685929 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:30:12.686117 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:30:12.686488 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:30:12.686494 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:30:12.686497 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:30:12.686854 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:30:12.687096 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0810 03:30:12.687450 (+ 354us) server_negotiation.cc:299] Negotiation successful
0810 03:30:12.687688 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260810 03:30:12.693439 20859 heartbeater.cc:503] Master 127.27.161.62:34901 was elected leader, sending a full tablet report...
I20260810 03:30:12.691726 20569 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36606:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 03:30:12.699896 20569 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 03:30:12.739266 21277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:30:12.713436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38030 (local address 127.27.161.2:36973)
0810 03:30:12.729649 (+ 16213us) server_negotiation.cc:206] Beginning negotiation
0810 03:30:12.729657 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 03:30:12.729679 (+ 22us) server_negotiation.cc:407] Connection header received
0810 03:30:12.729762 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:30:12.729768 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:30:12.729992 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:30:12.730188 (+ 196us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:30:12.731324 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:30:12.732768 (+ 1444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:30:12.735986 (+ 3218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:30:12.737566 (+ 1580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:30:12.737723 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:30:12.738014 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0810 03:30:12.738714 (+ 700us) server_negotiation.cc:299] Negotiation successful
0810 03:30:12.738943 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16018,"thread_start_us":15911,"threads_started":1}
I20260810 03:30:12.739295 21276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:30:12.713118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.1:38339 (local address 127.0.0.1:40960)
0810 03:30:12.728350 (+ 15232us) negotiation.cc:107] Waiting for socket to connect
0810 03:30:12.728371 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0810 03:30:12.728508 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:30:12.730058 (+ 1550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:30:12.730066 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:30:12.730100 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:30:12.730691 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:30:12.730723 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:30:12.732776 (+ 2053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:30:12.732782 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:30:12.736132 (+ 3350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:30:12.736149 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:30:12.738415 (+ 2266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:30:12.738421 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:30:12.738609 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:30:12.738641 (+ 32us) client_negotiation.cc:769] Sending connection context
0810 03:30:12.738716 (+ 75us) client_negotiation.cc:240] Negotiation successful
0810 03:30:12.738833 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15073,"mutex_wait_us":4,"thread_start_us":15470,"threads_started":1}
I20260810 03:30:12.739344 21279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:30:12.713881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40960 (local address 127.27.161.1:38339)
0810 03:30:12.729357 (+ 15476us) server_negotiation.cc:206] Beginning negotiation
0810 03:30:12.729366 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 03:30:12.729400 (+ 34us) server_negotiation.cc:407] Connection header received
0810 03:30:12.729524 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:30:12.729533 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:30:12.729775 (+ 242us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:30:12.730018 (+ 243us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:30:12.730979 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:30:12.732616 (+ 1637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:30:12.736423 (+ 3807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:30:12.737836 (+ 1413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:30:12.738016 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:30:12.738247 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0810 03:30:12.738853 (+ 606us) server_negotiation.cc:299] Negotiation successful
0810 03:30:12.739039 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15300,"thread_start_us":15095,"threads_started":1}
I20260810 03:30:12.740600 21275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:30:12.713045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.2:36973 (local address 127.0.0.1:38030)
0810 03:30:12.727188 (+ 14143us) negotiation.cc:107] Waiting for socket to connect
0810 03:30:12.727252 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0810 03:30:12.727517 (+ 265us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:30:12.730323 (+ 2806us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:30:12.730327 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:30:12.730349 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:30:12.731095 (+ 746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:30:12.731111 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:30:12.732917 (+ 1806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:30:12.732923 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:30:12.735716 (+ 2793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:30:12.735743 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:30:12.738141 (+ 2398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:30:12.738148 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:30:12.738355 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:30:12.738425 (+ 70us) client_negotiation.cc:769] Sending connection context
0810 03:30:12.738605 (+ 180us) client_negotiation.cc:240] Negotiation successful
0810 03:30:12.738828 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13764,"mutex_wait_us":206,"thread_start_us":13573,"threads_started":1}
I20260810 03:30:12.740971 21278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:30:12.713544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.3:44627 (local address 127.0.0.1:56522)
0810 03:30:12.729278 (+ 15734us) negotiation.cc:107] Waiting for socket to connect
0810 03:30:12.729299 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0810 03:30:12.729432 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:30:12.730411 (+ 979us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:30:12.730414 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:30:12.730431 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:30:12.731007 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:30:12.731020 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:30:12.733273 (+ 2253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:30:12.733282 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:30:12.735733 (+ 2451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:30:12.735743 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:30:12.737879 (+ 2136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:30:12.737887 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:30:12.739048 (+ 1161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:30:12.739082 (+ 34us) client_negotiation.cc:769] Sending connection context
0810 03:30:12.739178 (+ 96us) client_negotiation.cc:240] Negotiation successful
0810 03:30:12.739318 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15599,"mutex_wait_us":43,"thread_start_us":15450,"threads_started":1}
I20260810 03:30:12.741763 21280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:30:12.713996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56522 (local address 127.27.161.3:44627)
0810 03:30:12.723304 (+ 9308us) server_negotiation.cc:206] Beginning negotiation
0810 03:30:12.723316 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 03:30:12.729579 (+ 6263us) server_negotiation.cc:407] Connection header received
0810 03:30:12.729711 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:30:12.729718 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:30:12.729984 (+ 266us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:30:12.730177 (+ 193us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:30:12.731499 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:30:12.733071 (+ 1572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:30:12.735943 (+ 2872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:30:12.737566 (+ 1623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:30:12.737723 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:30:12.738024 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0810 03:30:12.741211 (+ 3187us) server_negotiation.cc:299] Negotiation successful
0810 03:30:12.741426 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9017,"thread_start_us":8822,"threads_started":1}
I20260810 03:30:12.743871 20753 tablet_service.cc:1467] Processing CreateTablet for tablet 2aca4f1321bd4bf0b5f757e209b5b17b (DEFAULT_TABLE table=test-table [id=eccf4386ab104bcba8ef5680175b6c95]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:30:12.745275 20753 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2aca4f1321bd4bf0b5f757e209b5b17b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:30:12.745014 20957 tablet_service.cc:1467] Processing CreateTablet for tablet 2aca4f1321bd4bf0b5f757e209b5b17b (DEFAULT_TABLE table=test-table [id=eccf4386ab104bcba8ef5680175b6c95]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:30:12.745385 21154 tablet_service.cc:1467] Processing CreateTablet for tablet 2aca4f1321bd4bf0b5f757e209b5b17b (DEFAULT_TABLE table=test-table [id=eccf4386ab104bcba8ef5680175b6c95]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:30:12.746589 20957 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2aca4f1321bd4bf0b5f757e209b5b17b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:30:12.746769 21154 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2aca4f1321bd4bf0b5f757e209b5b17b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:30:12.759548 21288 tablet_bootstrap.cc:492] T 2aca4f1321bd4bf0b5f757e209b5b17b P fac5507619954c158ebdd47a7d73fdc6: Bootstrap starting.
I20260810 03:30:12.763463 21291 tablet_bootstrap.cc:492] T 2aca4f1321bd4bf0b5f757e209b5b17b P c9a9fd5ce6b04fd5b5d2480a800f403c: Bootstrap starting.
I20260810 03:30:12.763592 21288 tablet_bootstrap.cc:654] T 2aca4f1321bd4bf0b5f757e209b5b17b P fac5507619954c158ebdd47a7d73fdc6: Neither blocks nor log segments found. Creating new log.
I20260810 03:30:12.764375 21290 tablet_bootstrap.cc:492] T 2aca4f1321bd4bf0b5f757e209b5b17b P f01667cb33c543839000d6eb9fb758d4: Bootstrap starting.
I20260810 03:30:12.766500 21288 log.cc:824] T 2aca4f1321bd4bf0b5f757e209b5b17b P fac5507619954c158ebdd47a7d73fdc6: Log is configured to *not* fsync() on all Append() calls
I20260810 03:30:12.768196 21290 tablet_bootstrap.cc:654] T 2aca4f1321bd4bf0b5f757e209b5b17b P f01667cb33c543839000d6eb9fb758d4: Neither blocks nor log segments found. Creating new log.
I20260810 03:30:12.768338 21291 tablet_bootstrap.cc:654] T 2aca4f1321bd4bf0b5f757e209b5b17b P c9a9fd5ce6b04fd5b5d2480a800f403c: Neither blocks nor log segments found. Creating new log.
I20260810 03:30:12.769402 21291 log.cc:824] T 2aca4f1321bd4bf0b5f757e209b5b17b P c9a9fd5ce6b04fd5b5d2480a800f403c: Log is configured to *not* fsync() on all Append() calls
I20260810 03:30:12.769547 21290 log.cc:824] T 2aca4f1321bd4bf0b5f757e209b5b17b P f01667cb33c543839000d6eb9fb758d4: Log is configured to *not* fsync() on all Append() calls
I20260810 03:30:12.770493 21288 tablet_bootstrap.cc:492] T 2aca4f1321bd4bf0b5f757e209b5b17b P fac5507619954c158ebdd47a7d73fdc6: No bootstrap required, opened a new log
I20260810 03:30:12.770783 21288 ts_tablet_manager.cc:1397] T 2aca4f1321bd4bf0b5f757e209b5b17b P fac5507619954c158ebdd47a7d73fdc6: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20260810 03:30:12.772462 21290 tablet_bootstrap.cc:492] T 2aca4f1321bd4bf0b5f757e209b5b17b P f01667cb33c543839000d6eb9fb758d4: No bootstrap required, opened a new log
I20260810 03:30:12.772716 21291 tablet_bootstrap.cc:492] T 2aca4f1321bd4bf0b5f757e209b5b17b P c9a9fd5ce6b04fd5b5d2480a800f403c: No bootstrap required, opened a new log
I20260810 03:30:12.772755 21290 ts_tablet_manager.cc:1397] T 2aca4f1321bd4bf0b5f757e209b5b17b P f01667cb33c543839000d6eb9fb758d4: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.006s
I20260810 03:30:12.773010 21291 ts_tablet_manager.cc:1397] T 2aca4f1321bd4bf0b5f757e209b5b17b P c9a9fd5ce6b04fd5b5d2480a800f403c: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.002s
I20260810 03:30:12.781019 21290 raft_consensus.cc:348] T 2aca4f1321bd4bf0b5f757e209b5b17b P f01667cb33c543839000d6eb9fb758d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f01667cb33c543839000d6eb9fb758d4" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 36973 } } peers { permanent_uuid: "fac5507619954c158ebdd47a7d73fdc6" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 38339 } } peers { permanent_uuid: "c9a9fd5ce6b04fd5b5d2480a800f403c" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 44627 } }
I20260810 03:30:12.781471 21290 raft_consensus.cc:374] T 2aca4f1321bd4bf0b5f757e209b5b17b P f01667cb33c543839000d6eb9fb758d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:30:12.781673 21290 raft_consensus.cc:729] T 2aca4f1321bd4bf0b5f757e209b5b17b P f01667cb33c543839000d6eb9fb758d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f01667cb33c543839000d6eb9fb758d4, State: Initialized, Role: FOLLOWER
I20260810 03:30:12.782476 21290 consensus_queue.cc:260] T 2aca4f1321bd4bf0b5f757e209b5b17b P f01667cb33c543839000d6eb9fb758d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f01667cb33c543839000d6eb9fb758d4" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 36973 } } peers { permanent_uuid: "fac5507619954c158ebdd47a7d73fdc6" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 38339 } } peers { permanent_uuid: "c9a9fd5ce6b04fd5b5d2480a800f403c" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 44627 } }
I20260810 03:30:12.782612 21288 raft_consensus.cc:348] T 2aca4f1321bd4bf0b5f757e209b5b17b P fac5507619954c158ebdd47a7d73fdc6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f01667cb33c543839000d6eb9fb758d4" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 36973 } } peers { permanent_uuid: "fac5507619954c158ebdd47a7d73fdc6" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 38339 } } peers { permanent_uuid: "c9a9fd5ce6b04fd5b5d2480a800f403c" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 44627 } }
I20260810 03:30:12.783198 21288 raft_consensus.cc:374] T 2aca4f1321bd4bf0b5f757e209b5b17b P fac5507619954c158ebdd47a7d73fdc6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:30:12.783380 21288 raft_consensus.cc:729] T 2aca4f1321bd4bf0b5f757e209b5b17b P fac5507619954c158ebdd47a7d73fdc6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fac5507619954c158ebdd47a7d73fdc6, State: Initialized, Role: FOLLOWER
I20260810 03:30:12.783718 21057 heartbeater.cc:503] Master 127.27.161.62:34901 was elected leader, sending a full tablet report...
I20260810 03:30:12.783891 21291 raft_consensus.cc:348] T 2aca4f1321bd4bf0b5f757e209b5b17b P c9a9fd5ce6b04fd5b5d2480a800f403c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f01667cb33c543839000d6eb9fb758d4" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 36973 } } peers { permanent_uuid: "fac5507619954c158ebdd47a7d73fdc6" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 38339 } } peers { permanent_uuid: "c9a9fd5ce6b04fd5b5d2480a800f403c" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 44627 } }
I20260810 03:30:12.784377 21291 raft_consensus.cc:374] T 2aca4f1321bd4bf0b5f757e209b5b17b P c9a9fd5ce6b04fd5b5d2480a800f403c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:30:12.784502 21291 raft_consensus.cc:729] T 2aca4f1321bd4bf0b5f757e209b5b17b P c9a9fd5ce6b04fd5b5d2480a800f403c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9a9fd5ce6b04fd5b5d2480a800f403c, State: Initialized, Role: FOLLOWER
I20260810 03:30:12.784258 21288 consensus_queue.cc:260] T 2aca4f1321bd4bf0b5f757e209b5b17b P fac5507619954c158ebdd47a7d73fdc6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f01667cb33c543839000d6eb9fb758d4" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 36973 } } peers { permanent_uuid: "fac5507619954c158ebdd47a7d73fdc6" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 38339 } } peers { permanent_uuid: "c9a9fd5ce6b04fd5b5d2480a800f403c" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 44627 } }
I20260810 03:30:12.784567 21290 ts_tablet_manager.cc:1428] T 2aca4f1321bd4bf0b5f757e209b5b17b P f01667cb33c543839000d6eb9fb758d4: Time spent starting tablet: real 0.012s user 0.011s sys 0.001s
I20260810 03:30:12.785151 21291 consensus_queue.cc:260] T 2aca4f1321bd4bf0b5f757e209b5b17b P c9a9fd5ce6b04fd5b5d2480a800f403c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f01667cb33c543839000d6eb9fb758d4" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 36973 } } peers { permanent_uuid: "fac5507619954c158ebdd47a7d73fdc6" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 38339 } } peers { permanent_uuid: "c9a9fd5ce6b04fd5b5d2480a800f403c" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 44627 } }
I20260810 03:30:12.786468 21288 ts_tablet_manager.cc:1428] T 2aca4f1321bd4bf0b5f757e209b5b17b P fac5507619954c158ebdd47a7d73fdc6: Time spent starting tablet: real 0.016s user 0.014s sys 0.000s
I20260810 03:30:12.789640 21257 heartbeater.cc:503] Master 127.27.161.62:34901 was elected leader, sending a full tablet report...
I20260810 03:30:12.790427 21291 ts_tablet_manager.cc:1428] T 2aca4f1321bd4bf0b5f757e209b5b17b P c9a9fd5ce6b04fd5b5d2480a800f403c: Time spent starting tablet: real 0.017s user 0.010s sys 0.007s
I20260810 03:30:12.798372 21300 raft_consensus.cc:482] T 2aca4f1321bd4bf0b5f757e209b5b17b P fac5507619954c158ebdd47a7d73fdc6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 03:30:12.798774 21300 raft_consensus.cc:504] T 2aca4f1321bd4bf0b5f757e209b5b17b P fac5507619954c158ebdd47a7d73fdc6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f01667cb33c543839000d6eb9fb758d4" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 36973 } } peers { permanent_uuid: "fac5507619954c158ebdd47a7d73fdc6" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 38339 } } peers { permanent_uuid: "c9a9fd5ce6b04fd5b5d2480a800f403c" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 44627 } }
I20260810 03:30:12.800705 21300 leader_election.cc:288] T 2aca4f1321bd4bf0b5f757e209b5b17b P fac5507619954c158ebdd47a7d73fdc6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f01667cb33c543839000d6eb9fb758d4 (127.27.161.2:36973), c9a9fd5ce6b04fd5b5d2480a800f403c (127.27.161.3:44627)
I20260810 03:30:12.807454 21280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:30:12.801550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.161.1:45179 (local address 127.27.161.3:44627)
0810 03:30:12.801745 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0810 03:30:12.801751 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:30:12.801991 (+ 240us) server_negotiation.cc:407] Connection header received
0810 03:30:12.802226 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:30:12.802231 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:30:12.802292 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:30:12.802367 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:30:12.802984 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:30:12.803712 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:30:12.805630 (+ 1918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:30:12.806494 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:30:12.806655 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:30:12.806736 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0810 03:30:12.807082 (+ 346us) server_negotiation.cc:299] Negotiation successful
0810 03:30:12.807212 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260810 03:30:12.807495 21277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:30:12.801114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.161.1:41307 (local address 127.27.161.2:36973)
0810 03:30:12.801333 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0810 03:30:12.801339 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:30:12.802094 (+ 755us) server_negotiation.cc:407] Connection header received
0810 03:30:12.802277 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:30:12.802283 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:30:12.802344 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:30:12.802419 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:30:12.803058 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:30:12.803771 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:30:12.805635 (+ 1864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:30:12.806516 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:30:12.806662 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:30:12.806753 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0810 03:30:12.807115 (+ 362us) server_negotiation.cc:299] Negotiation successful
0810 03:30:12.807248 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260810 03:30:12.807641 21303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:30:12.801002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.3:44627 (local address 127.27.161.1:45179)
0810 03:30:12.801877 (+ 875us) negotiation.cc:107] Waiting for socket to connect
0810 03:30:12.801896 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 03:30:12.802086 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:30:12.802403 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:30:12.802407 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:30:12.802432 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:30:12.802839 (+ 407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:30:12.802849 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:30:12.803851 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:30:12.803856 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:30:12.805465 (+ 1609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:30:12.805478 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:30:12.806736 (+ 1258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:30:12.806743 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:30:12.806895 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:30:12.806931 (+ 36us) client_negotiation.cc:769] Sending connection context
0810 03:30:12.807022 (+ 91us) client_negotiation.cc:240] Negotiation successful
0810 03:30:12.807162 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":713,"mutex_wait_us":45,"thread_start_us":743,"threads_started":1}
I20260810 03:30:12.807909 21305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:30:12.801095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.2:36973 (local address 127.27.161.1:41307)
0810 03:30:12.801991 (+ 896us) negotiation.cc:107] Waiting for socket to connect
0810 03:30:12.802008 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 03:30:12.802135 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:30:12.802445 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:30:12.802449 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:30:12.802473 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:30:12.802903 (+ 430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:30:12.802913 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:30:12.803903 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:30:12.803909 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:30:12.805466 (+ 1557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:30:12.805478 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:30:12.806793 (+ 1315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:30:12.806800 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:30:12.806949 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:30:12.806973 (+ 24us) client_negotiation.cc:769] Sending connection context
0810 03:30:12.807064 (+ 91us) client_negotiation.cc:240] Negotiation successful
0810 03:30:12.807197 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":763,"mutex_wait_us":37,"thread_start_us":487,"threads_started":1}
I20260810 03:30:12.808765 21185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2aca4f1321bd4bf0b5f757e209b5b17b" candidate_uuid: "fac5507619954c158ebdd47a7d73fdc6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9a9fd5ce6b04fd5b5d2480a800f403c" is_pre_election: true
I20260810 03:30:12.808789 20989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2aca4f1321bd4bf0b5f757e209b5b17b" candidate_uuid: "fac5507619954c158ebdd47a7d73fdc6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f01667cb33c543839000d6eb9fb758d4" is_pre_election: true
I20260810 03:30:12.809242 20989 raft_consensus.cc:2454] T 2aca4f1321bd4bf0b5f757e209b5b17b P f01667cb33c543839000d6eb9fb758d4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fac5507619954c158ebdd47a7d73fdc6 in term 0.
I20260810 03:30:12.809242 21185 raft_consensus.cc:2454] T 2aca4f1321bd4bf0b5f757e209b5b17b P c9a9fd5ce6b04fd5b5d2480a800f403c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fac5507619954c158ebdd47a7d73fdc6 in term 0.
I20260810 03:30:12.809867 20684 leader_election.cc:302] T 2aca4f1321bd4bf0b5f757e209b5b17b P fac5507619954c158ebdd47a7d73fdc6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c9a9fd5ce6b04fd5b5d2480a800f403c, fac5507619954c158ebdd47a7d73fdc6; no voters:
I20260810 03:30:12.810304 21300 raft_consensus.cc:2781] T 2aca4f1321bd4bf0b5f757e209b5b17b P fac5507619954c158ebdd47a7d73fdc6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 03:30:12.810465 21300 raft_consensus.cc:482] T 2aca4f1321bd4bf0b5f757e209b5b17b P fac5507619954c158ebdd47a7d73fdc6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 03:30:12.810575 21300 raft_consensus.cc:3037] T 2aca4f1321bd4bf0b5f757e209b5b17b P fac5507619954c158ebdd47a7d73fdc6 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:30:12.812763 21300 raft_consensus.cc:504] T 2aca4f1321bd4bf0b5f757e209b5b17b P fac5507619954c158ebdd47a7d73fdc6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f01667cb33c543839000d6eb9fb758d4" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 36973 } } peers { permanent_uuid: "fac5507619954c158ebdd47a7d73fdc6" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 38339 } } peers { permanent_uuid: "c9a9fd5ce6b04fd5b5d2480a800f403c" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 44627 } }
I20260810 03:30:12.813360 21300 leader_election.cc:288] T 2aca4f1321bd4bf0b5f757e209b5b17b P fac5507619954c158ebdd47a7d73fdc6 [CANDIDATE]: Term 1 election: Requested vote from peers f01667cb33c543839000d6eb9fb758d4 (127.27.161.2:36973), c9a9fd5ce6b04fd5b5d2480a800f403c (127.27.161.3:44627)
I20260810 03:30:12.813900 21185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2aca4f1321bd4bf0b5f757e209b5b17b" candidate_uuid: "fac5507619954c158ebdd47a7d73fdc6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9a9fd5ce6b04fd5b5d2480a800f403c"
I20260810 03:30:12.813899 20989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2aca4f1321bd4bf0b5f757e209b5b17b" candidate_uuid: "fac5507619954c158ebdd47a7d73fdc6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f01667cb33c543839000d6eb9fb758d4"
I20260810 03:30:12.814164 21185 raft_consensus.cc:3037] T 2aca4f1321bd4bf0b5f757e209b5b17b P c9a9fd5ce6b04fd5b5d2480a800f403c [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:30:12.814164 20989 raft_consensus.cc:3037] T 2aca4f1321bd4bf0b5f757e209b5b17b P f01667cb33c543839000d6eb9fb758d4 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:30:12.817428 20989 raft_consensus.cc:2454] T 2aca4f1321bd4bf0b5f757e209b5b17b P f01667cb33c543839000d6eb9fb758d4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fac5507619954c158ebdd47a7d73fdc6 in term 1.
I20260810 03:30:12.817430 21185 raft_consensus.cc:2454] T 2aca4f1321bd4bf0b5f757e209b5b17b P c9a9fd5ce6b04fd5b5d2480a800f403c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fac5507619954c158ebdd47a7d73fdc6 in term 1.
I20260810 03:30:12.817883 20687 leader_election.cc:302] T 2aca4f1321bd4bf0b5f757e209b5b17b P fac5507619954c158ebdd47a7d73fdc6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f01667cb33c543839000d6eb9fb758d4, fac5507619954c158ebdd47a7d73fdc6; no voters:
I20260810 03:30:12.818291 21300 raft_consensus.cc:2781] T 2aca4f1321bd4bf0b5f757e209b5b17b P fac5507619954c158ebdd47a7d73fdc6 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:30:12.818758 21300 raft_consensus.cc:686] T 2aca4f1321bd4bf0b5f757e209b5b17b P fac5507619954c158ebdd47a7d73fdc6 [term 1 LEADER]: Becoming Leader. State: Replica: fac5507619954c158ebdd47a7d73fdc6, State: Running, Role: LEADER
I20260810 03:30:12.819303 21300 consensus_queue.cc:237] T 2aca4f1321bd4bf0b5f757e209b5b17b P fac5507619954c158ebdd47a7d73fdc6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f01667cb33c543839000d6eb9fb758d4" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 36973 } } peers { permanent_uuid: "fac5507619954c158ebdd47a7d73fdc6" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 38339 } } peers { permanent_uuid: "c9a9fd5ce6b04fd5b5d2480a800f403c" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 44627 } }
I20260810 03:30:12.823869 20565 catalog_manager.cc:5495] T 2aca4f1321bd4bf0b5f757e209b5b17b P fac5507619954c158ebdd47a7d73fdc6 reported cstate change: term changed from 0 to 1, leader changed from <none> to fac5507619954c158ebdd47a7d73fdc6 (127.27.161.1). New cstate: current_term: 1 leader_uuid: "fac5507619954c158ebdd47a7d73fdc6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f01667cb33c543839000d6eb9fb758d4" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 36973 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fac5507619954c158ebdd47a7d73fdc6" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 38339 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c9a9fd5ce6b04fd5b5d2480a800f403c" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 44627 } health_report { overall_health: UNKNOWN } } }
I20260810 03:30:12.847489 21279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:30:12.842255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40968 (local address 127.27.161.1:38339)
0810 03:30:12.842457 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0810 03:30:12.842462 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:30:12.842481 (+ 19us) server_negotiation.cc:407] Connection header received
0810 03:30:12.842685 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:30:12.842689 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:30:12.842779 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:30:12.842924 (+ 145us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 03:30:12.843259 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:30:12.843964 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:30:12.845430 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:30:12.845841 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:30:12.845932 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:30:12.846476 (+ 544us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 03:30:12.846993 (+ 517us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 03:30:12.847102 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0810 03:30:12.847186 (+ 84us) server_negotiation.cc:299] Negotiation successful
0810 03:30:12.847272 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260810 03:30:12.856814 21185 raft_consensus.cc:1261] T 2aca4f1321bd4bf0b5f757e209b5b17b P c9a9fd5ce6b04fd5b5d2480a800f403c [term 1 FOLLOWER]: Refusing update from remote peer fac5507619954c158ebdd47a7d73fdc6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:30:12.856814 20989 raft_consensus.cc:1261] T 2aca4f1321bd4bf0b5f757e209b5b17b P f01667cb33c543839000d6eb9fb758d4 [term 1 FOLLOWER]: Refusing update from remote peer fac5507619954c158ebdd47a7d73fdc6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:30:12.858057 21300 consensus_queue.cc:1035] T 2aca4f1321bd4bf0b5f757e209b5b17b P fac5507619954c158ebdd47a7d73fdc6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9a9fd5ce6b04fd5b5d2480a800f403c" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 44627 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:30:12.858397 21306 consensus_queue.cc:1035] T 2aca4f1321bd4bf0b5f757e209b5b17b P fac5507619954c158ebdd47a7d73fdc6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f01667cb33c543839000d6eb9fb758d4" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 36973 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:30:12.877203 21309 mvcc.cc:205] Tried to move back new op lower bound from 7316818382249218048 to 7316818382111862784. Current Snapshot: MvccSnapshot[applied={T|T < 7316818382249218048}]
I20260810 03:30:12.887113 20565 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36606:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 03:30:12.887413 20565 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36606:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 03:30:12.891397 20565 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b85f2c98e9de4e5bbb5359e23783aaa0: Sending DeleteTablet for 3 replicas of tablet 2aca4f1321bd4bf0b5f757e209b5b17b
I20260810 03:30:12.892763 20957 tablet_service.cc:1514] Processing DeleteTablet for tablet 2aca4f1321bd4bf0b5f757e209b5b17b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:30:12 UTC) from {username='kudu', principal='kudu/127.27.161.62@KRBTEST.COM'} at 127.0.0.1:38030
I20260810 03:30:12.892778 20753 tablet_service.cc:1514] Processing DeleteTablet for tablet 2aca4f1321bd4bf0b5f757e209b5b17b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:30:12 UTC) from {username='kudu', principal='kudu/127.27.161.62@KRBTEST.COM'} at 127.0.0.1:40960
I20260810 03:30:12.893227 28292 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iBH5/build/asan/bin/kudu pid 20624
I20260810 03:30:12.894140 21154 tablet_service.cc:1514] Processing DeleteTablet for tablet 2aca4f1321bd4bf0b5f757e209b5b17b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:30:12 UTC) from {username='kudu', principal='kudu/127.27.161.62@KRBTEST.COM'} at 127.0.0.1:56522
I20260810 03:30:12.908094 21321 tablet_replica.cc:330] T 2aca4f1321bd4bf0b5f757e209b5b17b P c9a9fd5ce6b04fd5b5d2480a800f403c: stopping tablet replica
I20260810 03:30:12.908728 21321 raft_consensus.cc:2229] T 2aca4f1321bd4bf0b5f757e209b5b17b P c9a9fd5ce6b04fd5b5d2480a800f403c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 03:30:12.909432 21321 raft_consensus.cc:2258] T 2aca4f1321bd4bf0b5f757e209b5b17b P c9a9fd5ce6b04fd5b5d2480a800f403c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:30:12.912048 21321 ts_tablet_manager.cc:1905] T 2aca4f1321bd4bf0b5f757e209b5b17b P c9a9fd5ce6b04fd5b5d2480a800f403c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:30:12.919494 21321 ts_tablet_manager.cc:1918] T 2aca4f1321bd4bf0b5f757e209b5b17b P c9a9fd5ce6b04fd5b5d2480a800f403c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 03:30:12.919804 21321 log.cc:1196] T 2aca4f1321bd4bf0b5f757e209b5b17b P c9a9fd5ce6b04fd5b5d2480a800f403c: Deleting WAL directory at /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/minicluster-data/ts-2/wal/wals/2aca4f1321bd4bf0b5f757e209b5b17b
I20260810 03:30:12.920485 21321 ts_tablet_manager.cc:1939] T 2aca4f1321bd4bf0b5f757e209b5b17b P c9a9fd5ce6b04fd5b5d2480a800f403c: Deleting consensus metadata
I20260810 03:30:12.920668 21279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:30:12.894971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40980 (local address 127.27.161.1:38339)
0810 03:30:12.895158 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0810 03:30:12.895164 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:30:12.895195 (+ 31us) server_negotiation.cc:407] Connection header received
0810 03:30:12.895261 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:30:12.895267 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:30:12.895396 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:30:12.895510 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:30:12.897633 (+ 2123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:30:12.898973 (+ 1340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:30:12.901192 (+ 2219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:30:12.902068 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:30:12.902214 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:30:12.910459 (+ 8245us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:30:12.910476 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:30:12.910495 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:30:12.910556 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:30:12.914697 (+ 4141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:30:12.915758 (+ 1061us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:30:12.915768 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:30:12.915771 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:30:12.915988 (+ 217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:30:12.917323 (+ 1335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:30:12.917329 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:30:12.917332 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:30:12.917823 (+ 491us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:30:12.918198 (+ 375us) server_negotiation.cc:1035] Waiting for connection context
0810 03:30:12.919971 (+ 1773us) server_negotiation.cc:299] Negotiation successful
0810 03:30:12.920259 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260810 03:30:12.922437 20853 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:30:12.923075 20542 catalog_manager.cc:4841] TS c9a9fd5ce6b04fd5b5d2480a800f403c (127.27.161.3:44627): tablet 2aca4f1321bd4bf0b5f757e209b5b17b (table test-table [id=eccf4386ab104bcba8ef5680175b6c95]) successfully deleted
I20260810 03:30:12.925202 21319 tablet_replica.cc:330] T 2aca4f1321bd4bf0b5f757e209b5b17b P f01667cb33c543839000d6eb9fb758d4: stopping tablet replica
I20260810 03:30:12.925704 21319 raft_consensus.cc:2229] T 2aca4f1321bd4bf0b5f757e209b5b17b P f01667cb33c543839000d6eb9fb758d4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 03:30:12.926306 21319 raft_consensus.cc:2258] T 2aca4f1321bd4bf0b5f757e209b5b17b P f01667cb33c543839000d6eb9fb758d4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:30:12.928061 21319 ts_tablet_manager.cc:1905] T 2aca4f1321bd4bf0b5f757e209b5b17b P f01667cb33c543839000d6eb9fb758d4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:30:12.933104 21319 ts_tablet_manager.cc:1918] T 2aca4f1321bd4bf0b5f757e209b5b17b P f01667cb33c543839000d6eb9fb758d4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 03:30:12.933260 21319 log.cc:1196] T 2aca4f1321bd4bf0b5f757e209b5b17b P f01667cb33c543839000d6eb9fb758d4: Deleting WAL directory at /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/minicluster-data/ts-1/wal/wals/2aca4f1321bd4bf0b5f757e209b5b17b
I20260810 03:30:12.933727 21319 ts_tablet_manager.cc:1939] T 2aca4f1321bd4bf0b5f757e209b5b17b P f01667cb33c543839000d6eb9fb758d4: Deleting consensus metadata
I20260810 03:30:12.935904 20546 catalog_manager.cc:4841] TS f01667cb33c543839000d6eb9fb758d4 (127.27.161.2:36973): tablet 2aca4f1321bd4bf0b5f757e209b5b17b (table test-table [id=eccf4386ab104bcba8ef5680175b6c95]) successfully deleted
I20260810 03:30:13.023343 28292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iBH5/build/asan/bin/kudu with pid 20624
I20260810 03:30:13.050777 28292 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iBH5/build/asan/bin/kudu pid 20866
W20260810 03:30:13.050657 20543 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.161.1:38339 (error 108)
I20260810 03:30:13.052558 21278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:30:13.052115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.1:38339 (local address 127.0.0.1:40992)
0810 03:30:13.052281 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0810 03:30:13.052378 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.1:38339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260810 03:30:13.052839 20543 catalog_manager.cc:4584] TS fac5507619954c158ebdd47a7d73fdc6 (127.27.161.1:38339): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2aca4f1321bd4bf0b5f757e209b5b17b: Network error: Client connection negotiation failed: client connection to 127.27.161.1:38339: connect: Connection refused (error 111)
I20260810 03:30:13.069478 21277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:30:13.051905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38034 (local address 127.27.161.2:36973)
0810 03:30:13.052090 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0810 03:30:13.052096 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:30:13.052115 (+ 19us) server_negotiation.cc:407] Connection header received
0810 03:30:13.052203 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:30:13.052207 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:30:13.052290 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:30:13.052371 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:30:13.053890 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:30:13.054755 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:30:13.056025 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:30:13.056423 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:30:13.056512 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:30:13.063610 (+ 7098us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:30:13.063631 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:30:13.063655 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:30:13.063713 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:30:13.066713 (+ 3000us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:30:13.067474 (+ 761us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:30:13.067482 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:30:13.067484 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:30:13.067623 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:30:13.068056 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:30:13.068061 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:30:13.068065 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:30:13.068486 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:30:13.068736 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0810 03:30:13.069028 (+ 292us) server_negotiation.cc:299] Negotiation successful
0810 03:30:13.069214 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260810 03:30:13.070294 21051 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:30:13.106493 21278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:30:13.106011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.1:38339 (local address 127.0.0.1:40996)
0810 03:30:13.106236 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0810 03:30:13.106333 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.1:38339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
I20260810 03:30:13.107478 21278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:30:13.107142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.1:38339 (local address 127.0.0.1:41006)
0810 03:30:13.107294 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0810 03:30:13.107378 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.1:38339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260810 03:30:13.159641 28292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iBH5/build/asan/bin/kudu with pid 20866
I20260810 03:30:13.168215 21278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:30:13.167741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.1:38339 (local address 127.0.0.1:41008)
0810 03:30:13.167947 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0810 03:30:13.168055 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.1:38339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260810 03:30:13.169190 21278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:30:13.168903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.1:38339 (local address 127.0.0.1:41024)
0810 03:30:13.169019 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0810 03:30:13.169088 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.1:38339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260810 03:30:13.182941 28292 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iBH5/build/asan/bin/kudu pid 21061
I20260810 03:30:13.196393 21280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:30:13.183747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56536 (local address 127.27.161.3:44627)
0810 03:30:13.183929 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0810 03:30:13.183934 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:30:13.183955 (+ 21us) server_negotiation.cc:407] Connection header received
0810 03:30:13.184131 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:30:13.184135 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:30:13.184215 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:30:13.184302 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:30:13.185619 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:30:13.186410 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:30:13.187671 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:30:13.188077 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:30:13.188198 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:30:13.190838 (+ 2640us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:30:13.190853 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:30:13.190871 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:30:13.190908 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:30:13.193563 (+ 2655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:30:13.194316 (+ 753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:30:13.194324 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:30:13.194327 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:30:13.194453 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:30:13.194833 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:30:13.194837 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:30:13.194839 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:30:13.195198 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:30:13.195475 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0810 03:30:13.195819 (+ 344us) server_negotiation.cc:299] Negotiation successful
0810 03:30:13.196047 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260810 03:30:13.197034 21248 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:30:13.250267 21278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:30:13.249666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.1:38339 (local address 127.0.0.1:41026)
0810 03:30:13.249916 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0810 03:30:13.250059 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.1:38339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260810 03:30:13.251497 21278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:30:13.251034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.1:38339 (local address 127.0.0.1:41030)
0810 03:30:13.251267 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0810 03:30:13.251360 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.1:38339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113}
I20260810 03:30:13.288905 28292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iBH5/build/asan/bin/kudu with pid 21061
I20260810 03:30:13.315704 28292 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iBH5/build/asan/bin/kudu pid 20512
I20260810 03:30:13.316978 20613 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:30:13.436594 28292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iBH5/build/asan/bin/kudu with pid 20512
I20260810 03:30:13.436667 21278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:30:13.436058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.1:38339 (local address 127.0.0.1:41038)
0810 03:30:13.436377 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0810 03:30:13.436482 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.1:38339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166}
I20260810 03:30:33.467954 28292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iBH5/build/asan/bin/kudu
/tmp/dist-test-task24iBH5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.161.62:34901
--webserver_interface=127.27.161.62
--webserver_port=34355
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.62.keytab
--principal=kudu/127.27.161.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.161.20:42379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.161.62:34901
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:30:33.816912 21335 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:30:33.817217 21335 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:30:33.817317 21335 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:30:33.824522 21335 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260810 03:30:33.824645 21335 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260810 03:30:33.824693 21335 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260810 03:30:33.824746 21335 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:30:33.824800 21335 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:30:33.824870 21335 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:30:33.824915 21335 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:30:33.824970 21335 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:30:33.835283 21335 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.161.20:42379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.27.161.62:34901
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.62.keytab
--principal=kudu/127.27.161.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.161.62:34901
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.161.62
--webserver_port=34355
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:30:33.836649 21335 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:30:33.838004 21335 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:30:33.849622 21344 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:30:33.849589 21348 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:30:33.850461 21343 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:30:33.851095 21335 server_base.cc:1034] running on GCE node
I20260810 03:30:33.851917 21335 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:30:33.853366 21335 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:30:33.854610 21335 hybrid_clock.cc:648] HybridClock initialized: now 1786332633854528 us; error 37 us; skew 500 ppm
I20260810 03:30:33.854943 21335 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:30:33 dist-test-slave-vpvm krb5kdc[19667](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332633, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.161.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:30:33.860733 21335 init.cc:377] Logged in from keytab as kudu/127.27.161.62@KRBTEST.COM (short username kudu)
I20260810 03:30:33.862277 21335 webserver.cc:460] Webserver started at http://127.27.161.62:34355/ using document root <none> and password file <none>
I20260810 03:30:33.862988 21335 fs_manager.cc:359] Metadata directory not provided
I20260810 03:30:33.863091 21335 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:30:33.868599 21335 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260810 03:30:33.871554 21360 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:30:33.872884 21335 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260810 03:30:33.873062 21335 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/minicluster-data/master-0/data,/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/minicluster-data/master-0/wal
uuid: "b85f2c98e9de4e5bbb5359e23783aaa0"
format_stamp: "Formatted at 2026-08-10 03:29:47 on dist-test-slave-vpvm"
server_key: "1109bcb983d5b2e9d420ac696de1a21c"
server_key_iv: "d46666874395a5ac58653f3aa33db5cb"
server_key_version: "encryptionkey@0"
I20260810 03:30:33.874078 21335 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:30:33.893551 21335 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:30:33.897895 21335 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:30:33.898196 21335 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:30:33.915915 21335 rpc_server.cc:307] RPC server started. Bound to: 127.27.161.62:34901
I20260810 03:30:33.915946 21447 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.161.62:34901 every 8 connection(s)
I20260810 03:30:33.917371 21335 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/minicluster-data/master-0/data/info.pb
I20260810 03:30:33.920811 28292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iBH5/build/asan/bin/kudu as pid 21335
I20260810 03:30:33.921693 28292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iBH5/build/asan/bin/kudu
/tmp/dist-test-task24iBH5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.161.1:38339
--local_ip_for_outbound_sockets=127.27.161.1
--tserver_master_addrs=127.27.161.62:34901
--webserver_port=43831
--webserver_interface=127.27.161.1
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.1.keytab
--principal=kudu/127.27.161.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.161.20:42379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 03:30:33.924925 21449 sys_catalog.cc:263] Verifying existing consensus state
I20260810 03:30:33.928402 21449 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b85f2c98e9de4e5bbb5359e23783aaa0: Bootstrap starting.
I20260810 03:30:33.950064 21449 log.cc:824] T 00000000000000000000000000000000 P b85f2c98e9de4e5bbb5359e23783aaa0: Log is configured to *not* fsync() on all Append() calls
I20260810 03:30:33.968789 21449 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b85f2c98e9de4e5bbb5359e23783aaa0: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 03:30:33.969393 21449 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b85f2c98e9de4e5bbb5359e23783aaa0: Bootstrap complete.
I20260810 03:30:33.979211 21449 raft_consensus.cc:348] T 00000000000000000000000000000000 P b85f2c98e9de4e5bbb5359e23783aaa0 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b85f2c98e9de4e5bbb5359e23783aaa0" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 34901 } }
I20260810 03:30:33.980226 21449 raft_consensus.cc:729] T 00000000000000000000000000000000 P b85f2c98e9de4e5bbb5359e23783aaa0 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: b85f2c98e9de4e5bbb5359e23783aaa0, State: Initialized, Role: FOLLOWER
I20260810 03:30:33.980892 21449 consensus_queue.cc:260] T 00000000000000000000000000000000 P b85f2c98e9de4e5bbb5359e23783aaa0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b85f2c98e9de4e5bbb5359e23783aaa0" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 34901 } }
I20260810 03:30:33.981154 21449 raft_consensus.cc:388] T 00000000000000000000000000000000 P b85f2c98e9de4e5bbb5359e23783aaa0 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 03:30:33.981277 21449 raft_consensus.cc:482] T 00000000000000000000000000000000 P b85f2c98e9de4e5bbb5359e23783aaa0 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 03:30:33.981462 21449 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b85f2c98e9de4e5bbb5359e23783aaa0 [term 2 FOLLOWER]: Advancing to term 3
I20260810 03:30:33.985666 21449 raft_consensus.cc:504] T 00000000000000000000000000000000 P b85f2c98e9de4e5bbb5359e23783aaa0 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b85f2c98e9de4e5bbb5359e23783aaa0" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 34901 } }
I20260810 03:30:33.986228 21449 leader_election.cc:302] T 00000000000000000000000000000000 P b85f2c98e9de4e5bbb5359e23783aaa0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b85f2c98e9de4e5bbb5359e23783aaa0; no voters:
I20260810 03:30:33.986951 21449 leader_election.cc:288] T 00000000000000000000000000000000 P b85f2c98e9de4e5bbb5359e23783aaa0 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260810 03:30:33.987114 21453 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b85f2c98e9de4e5bbb5359e23783aaa0 [term 3 FOLLOWER]: Leader election won for term 3
I20260810 03:30:33.989081 21453 raft_consensus.cc:686] T 00000000000000000000000000000000 P b85f2c98e9de4e5bbb5359e23783aaa0 [term 3 LEADER]: Becoming Leader. State: Replica: b85f2c98e9de4e5bbb5359e23783aaa0, State: Running, Role: LEADER
I20260810 03:30:33.989277 21449 sys_catalog.cc:564] T 00000000000000000000000000000000 P b85f2c98e9de4e5bbb5359e23783aaa0 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:30:33.989703 21453 consensus_queue.cc:237] T 00000000000000000000000000000000 P b85f2c98e9de4e5bbb5359e23783aaa0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b85f2c98e9de4e5bbb5359e23783aaa0" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 34901 } }
I20260810 03:30:33.992353 21455 sys_catalog.cc:455] T 00000000000000000000000000000000 P b85f2c98e9de4e5bbb5359e23783aaa0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "b85f2c98e9de4e5bbb5359e23783aaa0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b85f2c98e9de4e5bbb5359e23783aaa0" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 34901 } } }
I20260810 03:30:33.992762 21455 sys_catalog.cc:458] T 00000000000000000000000000000000 P b85f2c98e9de4e5bbb5359e23783aaa0 [sys.catalog]: This master's current role is: LEADER
I20260810 03:30:33.992477 21456 sys_catalog.cc:455] T 00000000000000000000000000000000 P b85f2c98e9de4e5bbb5359e23783aaa0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b85f2c98e9de4e5bbb5359e23783aaa0. Latest consensus state: current_term: 3 leader_uuid: "b85f2c98e9de4e5bbb5359e23783aaa0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b85f2c98e9de4e5bbb5359e23783aaa0" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 34901 } } }
I20260810 03:30:33.992877 21456 sys_catalog.cc:458] T 00000000000000000000000000000000 P b85f2c98e9de4e5bbb5359e23783aaa0 [sys.catalog]: This master's current role is: LEADER
I20260810 03:30:34.000479 21466 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 03:30:34.007628 21466 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 03:30:34.008805 21466 catalog_manager.cc:1259] Loaded cluster ID: 49064dc1c4064843a5cd4d7e37ffd0d9
I20260810 03:30:34.008900 21466 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 03:30:34.011837 21466 catalog_manager.cc:1504] Loading token signing keys...
I20260810 03:30:34.013134 21466 catalog_manager.cc:5879] T 00000000000000000000000000000000 P b85f2c98e9de4e5bbb5359e23783aaa0: Loaded TSK: 0
I20260810 03:30:34.013265 21466 catalog_manager.cc:5879] T 00000000000000000000000000000000 P b85f2c98e9de4e5bbb5359e23783aaa0: Loaded TSK: 1
F20260810 03:30:34.014232 21466 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7ff8a85a99ad google::LogMessage::Fail() at ??:0
@ 0x7ff8a85ac88a google::LogMessage::SendToLog() at ??:0
@ 0x7ff8a85a94de google::LogMessage::Flush() at ??:0
@ 0x7ff8a85aaa19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7ff8ac30d7d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7ff8ac2fc73e kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7ff8bdfd81c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7ff8bdfd7783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7ff8be0632a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7ff8be0630c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7ff8be0ad5e9 std::function<>::operator()() at ??:0
@ 0x7ff8bdfdbcdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7ff8bdfd9c36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7ff8be061610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7ff8be061439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7ff8ab1f5345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7ff8ab1f891c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7ff8ab1f8799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7ff8ab1d3eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7ff8ae4ad6db start_thread at ??:0
@ 0x7ff8a744361f clone at ??:0
W20260810 03:30:34.346447 21451 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:30:34.346738 21451 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:30:34.346837 21451 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:30:34.354117 21451 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:30:34.354226 21451 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:30:34.354347 21451 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.161.1
I20260810 03:30:34.364082 21451 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.161.20:42379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.1.keytab
--principal=kudu/127.27.161.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.161.1:38339
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.161.1
--webserver_port=43831
--tserver_master_addrs=127.27.161.62:34901
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.161.1
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:30:34.365392 21451 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:30:34.366726 21451 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:30:34.379522 21526 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:30:34.379593 21524 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:30:34.379563 21523 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:30:34.380483 21451 server_base.cc:1034] running on GCE node
I20260810 03:30:34.381075 21451 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:30:34.382103 21451 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:30:34.383301 21451 hybrid_clock.cc:648] HybridClock initialized: now 1786332634383253 us; error 39 us; skew 500 ppm
I20260810 03:30:34.383625 21451 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:30:34 dist-test-slave-vpvm krb5kdc[19667](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332634, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.161.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:30:34.386638 21451 init.cc:377] Logged in from keytab as kudu/127.27.161.1@KRBTEST.COM (short username kudu)
I20260810 03:30:34.388120 21451 webserver.cc:460] Webserver started at http://127.27.161.1:43831/ using document root <none> and password file <none>
I20260810 03:30:34.388811 21451 fs_manager.cc:359] Metadata directory not provided
I20260810 03:30:34.388931 21451 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:30:34.394415 21451 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260810 03:30:34.397364 21536 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:30:34.398764 21451 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 03:30:34.398922 21451 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/minicluster-data/ts-0/data,/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/minicluster-data/ts-0/wal
uuid: "fac5507619954c158ebdd47a7d73fdc6"
format_stamp: "Formatted at 2026-08-10 03:29:48 on dist-test-slave-vpvm"
server_key: "a5851e1672c45620019b53ae23d94ea1"
server_key_iv: "ee5f60108cbc24f6eda769236ae9dc3e"
server_key_version: "encryptionkey@0"
I20260810 03:30:34.399890 21451 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:30:34.411079 21451 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:30:34.415124 21451 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:30:34.415374 21451 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:30:34.416388 21451 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:30:34.418251 21550 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260810 03:30:34.423914 21451 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260810 03:30:34.424043 21451 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.001s sys 0.000s
I20260810 03:30:34.424163 21451 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260810 03:30:34.427362 21451 ts_tablet_manager.cc:610] Registered 1 tablets
I20260810 03:30:34.427450 21451 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.002s sys 0.000s
I20260810 03:30:34.427772 21550 tablet_bootstrap.cc:492] T 2aca4f1321bd4bf0b5f757e209b5b17b P fac5507619954c158ebdd47a7d73fdc6: Bootstrap starting.
I20260810 03:30:34.454879 21550 log.cc:824] T 2aca4f1321bd4bf0b5f757e209b5b17b P fac5507619954c158ebdd47a7d73fdc6: Log is configured to *not* fsync() on all Append() calls
I20260810 03:30:34.467341 21451 rpc_server.cc:307] RPC server started. Bound to: 127.27.161.1:38339
I20260810 03:30:34.467361 21716 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.161.1:38339 every 8 connection(s)
I20260810 03:30:34.468962 21451 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/minicluster-data/ts-0/data/info.pb
I20260810 03:30:34.470743 21550 tablet_bootstrap.cc:492] T 2aca4f1321bd4bf0b5f757e209b5b17b P fac5507619954c158ebdd47a7d73fdc6: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 03:30:34.471397 21550 tablet_bootstrap.cc:492] T 2aca4f1321bd4bf0b5f757e209b5b17b P fac5507619954c158ebdd47a7d73fdc6: Bootstrap complete.
I20260810 03:30:34.473217 21550 ts_tablet_manager.cc:1397] T 2aca4f1321bd4bf0b5f757e209b5b17b P fac5507619954c158ebdd47a7d73fdc6: Time spent bootstrapping tablet: real 0.046s user 0.023s sys 0.018s
I20260810 03:30:34.474097 21720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:30:34.472029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:34901 (local address 127.27.161.1:32985)
0810 03:30:34.472878 (+ 849us) negotiation.cc:107] Waiting for socket to connect
0810 03:30:34.473144 (+ 266us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.62:34901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":605,"thread_start_us":383,"threads_started":1}
W20260810 03:30:34.474776 21717 heartbeater.cc:644] Failed to heartbeat to 127.27.161.62:34901 (0 consecutive failures): Network error: Failed to ping master at 127.27.161.62:34901: Client connection negotiation failed: client connection to 127.27.161.62:34901: connect: Connection refused (error 111)
I20260810 03:30:34.475638 21720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:30:34.475290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:34901 (local address 127.27.161.1:52759)
0810 03:30:34.475458 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0810 03:30:34.475529 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.62:34901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260810 03:30:34.476573 21720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:30:34.476228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:34901 (local address 127.27.161.1:38269)
0810 03:30:34.476356 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0810 03:30:34.476440 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.62:34901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260810 03:30:34.476819 21717 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260810 03:30:34.477391 28292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iBH5/build/asan/bin/kudu as pid 21451
I20260810 03:30:34.478024 28292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iBH5/build/asan/bin/kudu
/tmp/dist-test-task24iBH5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.161.2:36973
--local_ip_for_outbound_sockets=127.27.161.2
--tserver_master_addrs=127.27.161.62:34901
--webserver_port=43657
--webserver_interface=127.27.161.2
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.2.keytab
--principal=kudu/127.27.161.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.161.20:42379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 03:30:34.480603 21550 raft_consensus.cc:348] T 2aca4f1321bd4bf0b5f757e209b5b17b P fac5507619954c158ebdd47a7d73fdc6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f01667cb33c543839000d6eb9fb758d4" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 36973 } } peers { permanent_uuid: "fac5507619954c158ebdd47a7d73fdc6" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 38339 } } peers { permanent_uuid: "c9a9fd5ce6b04fd5b5d2480a800f403c" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 44627 } }
I20260810 03:30:34.481618 21550 raft_consensus.cc:729] T 2aca4f1321bd4bf0b5f757e209b5b17b P fac5507619954c158ebdd47a7d73fdc6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: fac5507619954c158ebdd47a7d73fdc6, State: Initialized, Role: FOLLOWER
I20260810 03:30:34.482317 21550 consensus_queue.cc:260] T 2aca4f1321bd4bf0b5f757e209b5b17b P fac5507619954c158ebdd47a7d73fdc6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f01667cb33c543839000d6eb9fb758d4" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 36973 } } peers { permanent_uuid: "fac5507619954c158ebdd47a7d73fdc6" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 38339 } } peers { permanent_uuid: "c9a9fd5ce6b04fd5b5d2480a800f403c" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 44627 } }
I20260810 03:30:34.483945 21550 ts_tablet_manager.cc:1428] T 2aca4f1321bd4bf0b5f757e209b5b17b P fac5507619954c158ebdd47a7d73fdc6: Time spent starting tablet: real 0.011s user 0.012s sys 0.001s
I20260810 03:30:34.484902 21720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:30:34.484531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:34901 (local address 127.27.161.1:47973)
0810 03:30:34.484696 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0810 03:30:34.484777 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.62:34901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260810 03:30:34.485872 21720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:30:34.485530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:34901 (local address 127.27.161.1:52001)
0810 03:30:34.485681 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0810 03:30:34.485748 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.62:34901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260810 03:30:34.821224 21728 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:30:34.821507 21728 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:30:34.821635 21728 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:30:34.829219 21728 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:30:34.829320 21728 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:30:34.829473 21728 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.161.2
I20260810 03:30:34.839140 21728 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.161.20:42379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.2.keytab
--principal=kudu/127.27.161.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.161.2:36973
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.161.2
--webserver_port=43657
--tserver_master_addrs=127.27.161.62:34901
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.161.2
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:30:34.840370 21728 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:30:34.841707 21728 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:30:34.854719 21738 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:30:34.854386 21734 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:30:34.855052 21736 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:30:34.855844 21728 server_base.cc:1034] running on GCE node
I20260810 03:30:34.856408 21728 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:30:34.857282 21728 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:30:34.858484 21728 hybrid_clock.cc:648] HybridClock initialized: now 1786332634858414 us; error 61 us; skew 500 ppm
I20260810 03:30:34.858800 21728 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:30:34 dist-test-slave-vpvm krb5kdc[19667](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332634, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.161.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:30:34.861824 21728 init.cc:377] Logged in from keytab as kudu/127.27.161.2@KRBTEST.COM (short username kudu)
I20260810 03:30:34.863269 21728 webserver.cc:460] Webserver started at http://127.27.161.2:43657/ using document root <none> and password file <none>
I20260810 03:30:34.863991 21728 fs_manager.cc:359] Metadata directory not provided
I20260810 03:30:34.864114 21728 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:30:34.869517 21728 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260810 03:30:34.872362 21753 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:30:34.873672 21728 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260810 03:30:34.873847 21728 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/minicluster-data/ts-1/data,/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/minicluster-data/ts-1/wal
uuid: "f01667cb33c543839000d6eb9fb758d4"
format_stamp: "Formatted at 2026-08-10 03:29:48 on dist-test-slave-vpvm"
server_key: "28983bd267833e90f784d217c84c1e21"
server_key_iv: "442c9269eed320d80b27b8851ba899c6"
server_key_version: "encryptionkey@0"
I20260810 03:30:34.874883 21728 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:30:34.893707 21728 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:30:34.897598 21728 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:30:34.897859 21728 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:30:34.898876 21728 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:30:34.900238 21728 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:30:34.900336 21728 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:30:34.900449 21728 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:30:34.900520 21728 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:30:34.933350 21728 rpc_server.cc:307] RPC server started. Bound to: 127.27.161.2:36973
I20260810 03:30:34.933403 21931 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.161.2:36973 every 8 connection(s)
I20260810 03:30:34.935304 21728 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/minicluster-data/ts-1/data/info.pb
I20260810 03:30:34.939438 28292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iBH5/build/asan/bin/kudu as pid 21728
I20260810 03:30:34.940197 28292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iBH5/build/asan/bin/kudu
/tmp/dist-test-task24iBH5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.161.3:44627
--local_ip_for_outbound_sockets=127.27.161.3
--tserver_master_addrs=127.27.161.62:34901
--webserver_port=42135
--webserver_interface=127.27.161.3
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.3.keytab
--principal=kudu/127.27.161.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.161.20:42379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 03:30:34.946120 21934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:30:34.943787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:34901 (local address 127.27.161.2:34003)
0810 03:30:34.944635 (+ 848us) negotiation.cc:107] Waiting for socket to connect
0810 03:30:34.944931 (+ 296us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.62:34901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":530,"thread_start_us":304,"threads_started":1}
W20260810 03:30:34.946875 21932 heartbeater.cc:644] Failed to heartbeat to 127.27.161.62:34901 (0 consecutive failures): Network error: Failed to ping master at 127.27.161.62:34901: Client connection negotiation failed: client connection to 127.27.161.62:34901: connect: Connection refused (error 111)
I20260810 03:30:34.947780 21934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:30:34.947381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:34901 (local address 127.27.161.2:55147)
0810 03:30:34.947549 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0810 03:30:34.947641 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.62:34901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260810 03:30:34.948741 21934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:30:34.948324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:34901 (local address 127.27.161.2:42129)
0810 03:30:34.948474 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0810 03:30:34.948563 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.62:34901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260810 03:30:34.948995 21932 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260810 03:30:35.296458 21935 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:30:35.296802 21935 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:30:35.296952 21935 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:30:35.307657 21935 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:30:35.307857 21935 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:30:35.308089 21935 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.161.3
I20260810 03:30:35.320596 21935 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.161.20:42379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.3.keytab
--principal=kudu/127.27.161.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.161.3:44627
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.161.3
--webserver_port=42135
--tserver_master_addrs=127.27.161.62:34901
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.161.3
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:30:35.321914 21935 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:30:35.323284 21935 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:30:35.336810 21952 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:30:35.336819 21949 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:30:35.336997 21948 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:30:35.338042 21935 server_base.cc:1034] running on GCE node
I20260810 03:30:35.338783 21935 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:30:35.339797 21935 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:30:35.341017 21935 hybrid_clock.cc:648] HybridClock initialized: now 1786332635340951 us; error 46 us; skew 500 ppm
I20260810 03:30:35.341357 21935 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:30:35 dist-test-slave-vpvm krb5kdc[19667](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332635, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.161.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:30:35.344724 21935 init.cc:377] Logged in from keytab as kudu/127.27.161.3@KRBTEST.COM (short username kudu)
I20260810 03:30:35.346402 21935 webserver.cc:460] Webserver started at http://127.27.161.3:42135/ using document root <none> and password file <none>
I20260810 03:30:35.347170 21935 fs_manager.cc:359] Metadata directory not provided
I20260810 03:30:35.347311 21935 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:30:35.353103 21935 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260810 03:30:35.356055 21962 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:30:35.357381 21935 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 03:30:35.357594 21935 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/minicluster-data/ts-2/data,/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/minicluster-data/ts-2/wal
uuid: "c9a9fd5ce6b04fd5b5d2480a800f403c"
format_stamp: "Formatted at 2026-08-10 03:29:49 on dist-test-slave-vpvm"
server_key: "eed21fa6d563145c0f2de57944d4bf26"
server_key_iv: "c70ac530c2efd77a7ba72de0ce6a5193"
server_key_version: "encryptionkey@0"
I20260810 03:30:35.358736 21935 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:30:35.375758 21935 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:30:35.379828 21935 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:30:35.380115 21935 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:30:35.381155 21935 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:30:35.382670 21935 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:30:35.382777 21935 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:30:35.382891 21935 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:30:35.382983 21935 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:30:35.417037 21935 rpc_server.cc:307] RPC server started. Bound to: 127.27.161.3:44627
I20260810 03:30:35.417086 22133 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.161.3:44627 every 8 connection(s)
I20260810 03:30:35.418650 21935 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786332336319544-28292-0/minicluster-data/ts-2/data/info.pb
I20260810 03:30:35.423491 28292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iBH5/build/asan/bin/kudu as pid 21935
I20260810 03:30:35.423509 22139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:30:35.421634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:34901 (local address 127.27.161.3:46367)
0810 03:30:35.422401 (+ 767us) negotiation.cc:107] Waiting for socket to connect
0810 03:30:35.422633 (+ 232us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.62:34901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":303,"threads_started":1}
W20260810 03:30:35.424207 22134 heartbeater.cc:644] Failed to heartbeat to 127.27.161.62:34901 (0 consecutive failures): Network error: Failed to ping master at 127.27.161.62:34901: Client connection negotiation failed: client connection to 127.27.161.62:34901: connect: Connection refused (error 111)
I20260810 03:30:35.425068 22139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:30:35.424725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:34901 (local address 127.27.161.3:52281)
0810 03:30:35.424880 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0810 03:30:35.424963 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.62:34901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260810 03:30:35.425937 22139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:30:35.425623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:34901 (local address 127.27.161.3:49415)
0810 03:30:35.425751 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0810 03:30:35.425826 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.62:34901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260810 03:30:35.426208 22134 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260810 03:30:35.487455 22141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:30:35.486718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:34901 (local address 127.27.161.1:38295)
0810 03:30:35.487171 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0810 03:30:35.487255 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.62:34901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":204,"threads_started":1}
I20260810 03:30:35.950383 22142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:30:35.949509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:34901 (local address 127.27.161.2:51921)
0810 03:30:35.950026 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0810 03:30:35.950137 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.62:34901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":228,"threads_started":1}
I20260810 03:30:36.133348 22143 raft_consensus.cc:482] T 2aca4f1321bd4bf0b5f757e209b5b17b P fac5507619954c158ebdd47a7d73fdc6 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 03:30:36.133626 22143 raft_consensus.cc:504] T 2aca4f1321bd4bf0b5f757e209b5b17b P fac5507619954c158ebdd47a7d73fdc6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f01667cb33c543839000d6eb9fb758d4" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 36973 } } peers { permanent_uuid: "fac5507619954c158ebdd47a7d73fdc6" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 38339 } } peers { permanent_uuid: "c9a9fd5ce6b04fd5b5d2480a800f403c" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 44627 } }
I20260810 03:30:36.135330 22143 leader_election.cc:288] T 2aca4f1321bd4bf0b5f757e209b5b17b P fac5507619954c158ebdd47a7d73fdc6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f01667cb33c543839000d6eb9fb758d4 (127.27.161.2:36973), c9a9fd5ce6b04fd5b5d2480a800f403c (127.27.161.3:44627)
Aug 10 03:30:36 dist-test-slave-vpvm krb5kdc[19667](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332634, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.161.1@KRBTEST.COM for kudu/127.27.161.3@KRBTEST.COM
Aug 10 03:30:36 dist-test-slave-vpvm krb5kdc[19667](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332634, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.161.1@KRBTEST.COM for kudu/127.27.161.2@KRBTEST.COM
I20260810 03:30:36.175053 22147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:30:36.135959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.161.1:35455 (local address 127.27.161.3:44627)
0810 03:30:36.149200 (+ 13241us) server_negotiation.cc:206] Beginning negotiation
0810 03:30:36.149215 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 03:30:36.149818 (+ 603us) server_negotiation.cc:407] Connection header received
0810 03:30:36.152604 (+ 2786us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:30:36.152617 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:30:36.153124 (+ 507us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:30:36.153478 (+ 354us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:30:36.156518 (+ 3040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:30:36.158473 (+ 1955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:30:36.161480 (+ 3007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:30:36.162572 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:30:36.162770 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:30:36.167396 (+ 4626us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:30:36.167405 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:30:36.167425 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:30:36.167476 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:30:36.171308 (+ 3832us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:30:36.172044 (+ 736us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:30:36.172051 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:30:36.172056 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:30:36.172248 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:30:36.172701 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:30:36.172704 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:30:36.172707 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:30:36.173247 (+ 540us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:30:36.173418 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0810 03:30:36.174268 (+ 850us) server_negotiation.cc:299] Negotiation successful
0810 03:30:36.174618 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13088,"thread_start_us":12997,"threads_started":1}
I20260810 03:30:36.175062 22148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:30:36.135959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.161.1:59753 (local address 127.27.161.2:36973)
0810 03:30:36.148565 (+ 12606us) server_negotiation.cc:206] Beginning negotiation
0810 03:30:36.148588 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0810 03:30:36.149819 (+ 1231us) server_negotiation.cc:407] Connection header received
0810 03:30:36.152697 (+ 2878us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:30:36.152718 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:30:36.153099 (+ 381us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:30:36.153474 (+ 375us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:30:36.156518 (+ 3044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:30:36.158473 (+ 1955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:30:36.161480 (+ 3007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:30:36.162572 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:30:36.162770 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:30:36.167615 (+ 4845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:30:36.167624 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:30:36.167638 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:30:36.167694 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:30:36.171308 (+ 3614us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:30:36.172473 (+ 1165us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:30:36.172480 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:30:36.172485 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:30:36.172641 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:30:36.173012 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:30:36.173015 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:30:36.173017 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:30:36.173502 (+ 485us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:30:36.173665 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0810 03:30:36.174276 (+ 611us) server_negotiation.cc:299] Negotiation successful
0810 03:30:36.174662 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12367,"thread_start_us":12652,"threads_started":1}
I20260810 03:30:36.175209 22144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:30:36.135561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.3:44627 (local address 127.27.161.1:35455)
0810 03:30:36.149599 (+ 14038us) negotiation.cc:107] Waiting for socket to connect
0810 03:30:36.149631 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0810 03:30:36.150288 (+ 657us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:30:36.153996 (+ 3708us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:30:36.154001 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:30:36.155167 (+ 1166us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:30:36.156278 (+ 1111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:30:36.156308 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:30:36.158725 (+ 2417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:30:36.158732 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:30:36.161314 (+ 2582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:30:36.161335 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:30:36.162862 (+ 1527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:30:36.162872 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:30:36.163297 (+ 425us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:30:36.164222 (+ 925us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:30:36.164276 (+ 54us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:30:36.167063 (+ 2787us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:30:36.171465 (+ 4402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:30:36.171472 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:30:36.171485 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:30:36.171910 (+ 425us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:30:36.172381 (+ 471us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:30:36.172386 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:30:36.172390 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:30:36.172577 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:30:36.173397 (+ 820us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:30:36.173405 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:30:36.173690 (+ 285us) client_negotiation.cc:769] Sending connection context
0810 03:30:36.173981 (+ 291us) client_negotiation.cc:240] Negotiation successful
0810 03:30:36.174730 (+ 749us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13899,"mutex_wait_us":601,"spinlock_wait_cycles":53376,"thread_start_us":13730,"threads_started":1}
I20260810 03:30:36.175129 22146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:30:36.135561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.2:36973 (local address 127.27.161.1:59753)
0810 03:30:36.149679 (+ 14118us) negotiation.cc:107] Waiting for socket to connect
0810 03:30:36.149704 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0810 03:30:36.150279 (+ 575us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:30:36.153652 (+ 3373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:30:36.153663 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:30:36.155167 (+ 1504us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:30:36.156278 (+ 1111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:30:36.156308 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:30:36.158729 (+ 2421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:30:36.158733 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:30:36.161314 (+ 2581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:30:36.161335 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:30:36.162890 (+ 1555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:30:36.162898 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:30:36.163290 (+ 392us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:30:36.164222 (+ 932us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:30:36.164276 (+ 54us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:30:36.167344 (+ 3068us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:30:36.172016 (+ 4672us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:30:36.172019 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:30:36.172021 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:30:36.172328 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:30:36.172769 (+ 441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:30:36.172773 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:30:36.172775 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:30:36.172900 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:30:36.173640 (+ 740us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:30:36.173645 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:30:36.173800 (+ 155us) client_negotiation.cc:769] Sending connection context
0810 03:30:36.173979 (+ 179us) client_negotiation.cc:240] Negotiation successful
0810 03:30:36.174572 (+ 593us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14017,"mutex_wait_us":438,"spinlock_wait_cycles":9984,"thread_start_us":13743,"threads_started":1}
W20260810 03:30:36.178515 21539 leader_election.cc:341] T 2aca4f1321bd4bf0b5f757e209b5b17b P fac5507619954c158ebdd47a7d73fdc6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c9a9fd5ce6b04fd5b5d2480a800f403c (127.27.161.3:44627): Not found: Tablet not found: 2aca4f1321bd4bf0b5f757e209b5b17b
W20260810 03:30:36.178828 21546 leader_election.cc:341] T 2aca4f1321bd4bf0b5f757e209b5b17b P fac5507619954c158ebdd47a7d73fdc6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f01667cb33c543839000d6eb9fb758d4 (127.27.161.2:36973): Not found: Tablet not found: 2aca4f1321bd4bf0b5f757e209b5b17b
I20260810 03:30:36.178985 21546 leader_election.cc:302] T 2aca4f1321bd4bf0b5f757e209b5b17b P fac5507619954c158ebdd47a7d73fdc6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fac5507619954c158ebdd47a7d73fdc6; no voters: c9a9fd5ce6b04fd5b5d2480a800f403c, f01667cb33c543839000d6eb9fb758d4
I20260810 03:30:36.179395 22143 raft_consensus.cc:2726] T 2aca4f1321bd4bf0b5f757e209b5b17b P fac5507619954c158ebdd47a7d73fdc6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 03:30:36.427561 22149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:30:36.426713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:34901 (local address 127.27.161.3:46701)
0810 03:30:36.427171 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0810 03:30:36.427320 (+ 149us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.62:34901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":202,"threads_started":1}
I20260810 03:30:36.488734 22146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:30:36.488312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:34901 (local address 127.27.161.1:56507)
0810 03:30:36.488490 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0810 03:30:36.488587 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.62:34901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260810 03:30:36.952008 22150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:30:36.951274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:34901 (local address 127.27.161.2:36637)
0810 03:30:36.951754 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0810 03:30:36.951844 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.62:34901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":250,"threads_started":1}
I20260810 03:30:37.429566 22151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:30:37.428744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:34901 (local address 127.27.161.3:57453)
0810 03:30:37.429291 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0810 03:30:37.429389 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.62:34901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":296,"threads_started":1}
I20260810 03:30:37.490631 22154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:30:37.489666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:34901 (local address 127.27.161.1:56785)
0810 03:30:37.490340 (+ 674us) negotiation.cc:107] Waiting for socket to connect
0810 03:30:37.490432 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.62:34901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":565,"thread_start_us":329,"threads_started":1}
I20260810 03:30:37.954095 22155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:30:37.953028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:34901 (local address 127.27.161.2:52997)
0810 03:30:37.953677 (+ 649us) negotiation.cc:107] Waiting for socket to connect
0810 03:30:37.953831 (+ 154us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.62:34901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":280,"threads_started":1}
I20260810 03:30:38.339075 22156 raft_consensus.cc:482] T 2aca4f1321bd4bf0b5f757e209b5b17b P fac5507619954c158ebdd47a7d73fdc6 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 03:30:38.339301 22156 raft_consensus.cc:504] T 2aca4f1321bd4bf0b5f757e209b5b17b P fac5507619954c158ebdd47a7d73fdc6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f01667cb33c543839000d6eb9fb758d4" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 36973 } } peers { permanent_uuid: "fac5507619954c158ebdd47a7d73fdc6" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 38339 } } peers { permanent_uuid: "c9a9fd5ce6b04fd5b5d2480a800f403c" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 44627 } }
I20260810 03:30:38.340076 22156 leader_election.cc:288] T 2aca4f1321bd4bf0b5f757e209b5b17b P fac5507619954c158ebdd47a7d73fdc6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f01667cb33c543839000d6eb9fb758d4 (127.27.161.2:36973), c9a9fd5ce6b04fd5b5d2480a800f403c (127.27.161.3:44627)
W20260810 03:30:38.341156 21546 leader_election.cc:341] T 2aca4f1321bd4bf0b5f757e209b5b17b P fac5507619954c158ebdd47a7d73fdc6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f01667cb33c543839000d6eb9fb758d4 (127.27.161.2:36973): Not found: Tablet not found: 2aca4f1321bd4bf0b5f757e209b5b17b
W20260810 03:30:38.341336 21539 leader_election.cc:341] T 2aca4f1321bd4bf0b5f757e209b5b17b P fac5507619954c158ebdd47a7d73fdc6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c9a9fd5ce6b04fd5b5d2480a800f403c (127.27.161.3:44627): Not found: Tablet not found: 2aca4f1321bd4bf0b5f757e209b5b17b
I20260810 03:30:38.341455 21539 leader_election.cc:302] T 2aca4f1321bd4bf0b5f757e209b5b17b P fac5507619954c158ebdd47a7d73fdc6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fac5507619954c158ebdd47a7d73fdc6; no voters: c9a9fd5ce6b04fd5b5d2480a800f403c, f01667cb33c543839000d6eb9fb758d4
I20260810 03:30:38.341727 22156 raft_consensus.cc:2726] T 2aca4f1321bd4bf0b5f757e209b5b17b P fac5507619954c158ebdd47a7d73fdc6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 03:30:38.431150 22157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:30:38.430461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:34901 (local address 127.27.161.3:50847)
0810 03:30:38.430905 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0810 03:30:38.430989 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.62:34901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":232,"threads_started":1}
I20260810 03:30:38.492484 22158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:30:38.491637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:34901 (local address 127.27.161.1:41069)
0810 03:30:38.492175 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0810 03:30:38.492288 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.62:34901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":261,"threads_started":1}
I20260810 03:30:38.956004 22159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:30:38.955147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:34901 (local address 127.27.161.2:34473)
0810 03:30:38.955702 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0810 03:30:38.955798 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.62:34901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":264,"threads_started":1}
I20260810 03:30:39.433135 22160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:30:39.432266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:34901 (local address 127.27.161.3:45631)
0810 03:30:39.432836 (+ 570us) negotiation.cc:107] Waiting for socket to connect
0810 03:30:39.432935 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.62:34901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":272,"threads_started":1}
I20260810 03:30:39.494616 22161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:30:39.493790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:34901 (local address 127.27.161.1:48511)
0810 03:30:39.494324 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0810 03:30:39.494417 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.62:34901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":287,"threads_started":1}
I20260810 03:30:39.958070 22162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:30:39.957106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:34901 (local address 127.27.161.2:40691)
0810 03:30:39.957736 (+ 630us) negotiation.cc:107] Waiting for socket to connect
0810 03:30:39.957837 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.62:34901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":281,"threads_started":1}
I20260810 03:30:40.119520 22163 raft_consensus.cc:482] T 2aca4f1321bd4bf0b5f757e209b5b17b P fac5507619954c158ebdd47a7d73fdc6 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 03:30:40.119786 22163 raft_consensus.cc:504] T 2aca4f1321bd4bf0b5f757e209b5b17b P fac5507619954c158ebdd47a7d73fdc6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f01667cb33c543839000d6eb9fb758d4" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 36973 } } peers { permanent_uuid: "fac5507619954c158ebdd47a7d73fdc6" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 38339 } } peers { permanent_uuid: "c9a9fd5ce6b04fd5b5d2480a800f403c" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 44627 } }
I20260810 03:30:40.120456 22163 leader_election.cc:288] T 2aca4f1321bd4bf0b5f757e209b5b17b P fac5507619954c158ebdd47a7d73fdc6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f01667cb33c543839000d6eb9fb758d4 (127.27.161.2:36973), c9a9fd5ce6b04fd5b5d2480a800f403c (127.27.161.3:44627)
W20260810 03:30:40.121461 21539 leader_election.cc:341] T 2aca4f1321bd4bf0b5f757e209b5b17b P fac5507619954c158ebdd47a7d73fdc6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c9a9fd5ce6b04fd5b5d2480a800f403c (127.27.161.3:44627): Not found: Tablet not found: 2aca4f1321bd4bf0b5f757e209b5b17b
W20260810 03:30:40.121630 21546 leader_election.cc:341] T 2aca4f1321bd4bf0b5f757e209b5b17b P fac5507619954c158ebdd47a7d73fdc6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f01667cb33c543839000d6eb9fb758d4 (127.27.161.2:36973): Not found: Tablet not found: 2aca4f1321bd4bf0b5f757e209b5b17b
I20260810 03:30:40.121774 21546 leader_election.cc:302] T 2aca4f1321bd4bf0b5f757e209b5b17b P fac5507619954c158ebdd47a7d73fdc6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fac5507619954c158ebdd47a7d73fdc6; no voters: c9a9fd5ce6b04fd5b5d2480a800f403c, f01667cb33c543839000d6eb9fb758d4
I20260810 03:30:40.122042 22163 raft_consensus.cc:2726] T 2aca4f1321bd4bf0b5f757e209b5b17b P fac5507619954c158ebdd47a7d73fdc6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 03:30:40.434898 22164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:30:40.434178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:34901 (local address 127.27.161.3:53625)
0810 03:30:40.434623 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0810 03:30:40.434724 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.62:34901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":230,"threads_started":1}
I20260810 03:30:40.496316 22167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:30:40.495575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:34901 (local address 127.27.161.1:57125)
0810 03:30:40.496042 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0810 03:30:40.496128 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.62:34901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":232,"threads_started":1}
I20260810 03:30:40.959824 22168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:30:40.959059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:34901 (local address 127.27.161.2:60753)
0810 03:30:40.959527 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0810 03:30:40.959629 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.62:34901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":234,"threads_started":1}
I20260810 03:30:41.436542 22170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:30:41.435780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:34901 (local address 127.27.161.3:35605)
0810 03:30:41.436274 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0810 03:30:41.436359 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.62:34901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":255,"threads_started":1}
I20260810 03:30:41.498003 22171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:30:41.497208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:34901 (local address 127.27.161.1:44919)
0810 03:30:41.497671 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0810 03:30:41.497766 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.62:34901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":248,"threads_started":1}
I20260810 03:30:41.961733 22175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:30:41.960780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:34901 (local address 127.27.161.2:60593)
0810 03:30:41.961399 (+ 619us) negotiation.cc:107] Waiting for socket to connect
0810 03:30:41.961513 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.62:34901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":493,"thread_start_us":275,"threads_started":1}
I20260810 03:30:42.390627 22177 raft_consensus.cc:482] T 2aca4f1321bd4bf0b5f757e209b5b17b P fac5507619954c158ebdd47a7d73fdc6 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 03:30:42.390836 22177 raft_consensus.cc:504] T 2aca4f1321bd4bf0b5f757e209b5b17b P fac5507619954c158ebdd47a7d73fdc6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f01667cb33c543839000d6eb9fb758d4" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 36973 } } peers { permanent_uuid: "fac5507619954c158ebdd47a7d73fdc6" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 38339 } } peers { permanent_uuid: "c9a9fd5ce6b04fd5b5d2480a800f403c" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 44627 } }
I20260810 03:30:42.391506 22177 leader_election.cc:288] T 2aca4f1321bd4bf0b5f757e209b5b17b P fac5507619954c158ebdd47a7d73fdc6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f01667cb33c543839000d6eb9fb758d4 (127.27.161.2:36973), c9a9fd5ce6b04fd5b5d2480a800f403c (127.27.161.3:44627)
W20260810 03:30:42.392570 21539 leader_election.cc:341] T 2aca4f1321bd4bf0b5f757e209b5b17b P fac5507619954c158ebdd47a7d73fdc6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c9a9fd5ce6b04fd5b5d2480a800f403c (127.27.161.3:44627): Not found: Tablet not found: 2aca4f1321bd4bf0b5f757e209b5b17b
W20260810 03:30:42.392758 21546 leader_election.cc:341] T 2aca4f1321bd4bf0b5f757e209b5b17b P fac5507619954c158ebdd47a7d73fdc6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f01667cb33c543839000d6eb9fb758d4 (127.27.161.2:36973): Not found: Tablet not found: 2aca4f1321bd4bf0b5f757e209b5b17b
I20260810 03:30:42.392880 21546 leader_election.cc:302] T 2aca4f1321bd4bf0b5f757e209b5b17b P fac5507619954c158ebdd47a7d73fdc6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fac5507619954c158ebdd47a7d73fdc6; no voters: c9a9fd5ce6b04fd5b5d2480a800f403c, f01667cb33c543839000d6eb9fb758d4
I20260810 03:30:42.393116 22177 raft_consensus.cc:2726] T 2aca4f1321bd4bf0b5f757e209b5b17b P fac5507619954c158ebdd47a7d73fdc6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 03:30:42.438522 22178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:30:42.437592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:34901 (local address 127.27.161.3:56405)
0810 03:30:42.438220 (+ 628us) negotiation.cc:107] Waiting for socket to connect
0810 03:30:42.438328 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.62:34901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":510,"thread_start_us":267,"threads_started":1}
I20260810 03:30:42.499850 22179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:30:42.499060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:34901 (local address 127.27.161.1:36577)
0810 03:30:42.499553 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0810 03:30:42.499635 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.62:34901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":260,"threads_started":1}
I20260810 03:30:42.963692 22180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:30:42.962782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:34901 (local address 127.27.161.2:56421)
0810 03:30:42.963291 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0810 03:30:42.963471 (+ 180us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.62:34901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":239,"threads_started":1}
I20260810 03:30:43.440261 22181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:30:43.439554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:34901 (local address 127.27.161.3:34385)
0810 03:30:43.439987 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0810 03:30:43.440074 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.62:34901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":212,"threads_started":1}
I20260810 03:30:43.501770 22183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:30:43.500896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:34901 (local address 127.27.161.1:53497)
0810 03:30:43.501425 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0810 03:30:43.501536 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.62:34901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":269,"threads_started":1}
I20260810 03:30:43.965459 22186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:30:43.964698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:34901 (local address 127.27.161.2:35107)
0810 03:30:43.965190 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0810 03:30:43.965286 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.62:34901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":222,"threads_started":1}
I20260810 03:30:44.441918 22187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:30:44.441196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:34901 (local address 127.27.161.3:50211)
0810 03:30:44.441667 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0810 03:30:44.441756 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.62:34901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":223,"threads_started":1}
I20260810 03:30:44.503515 22189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:30:44.502788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:34901 (local address 127.27.161.1:58001)
0810 03:30:44.503237 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0810 03:30:44.503342 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.62:34901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":232,"threads_started":1}
I20260810 03:30:44.967252 22190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:30:44.966460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:34901 (local address 127.27.161.2:59121)
0810 03:30:44.966948 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0810 03:30:44.967060 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.62:34901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":259,"threads_started":1}
I20260810 03:30:45.443848 22193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:30:45.442994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:34901 (local address 127.27.161.3:52865)
0810 03:30:45.443529 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0810 03:30:45.443631 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.62:34901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":265,"threads_started":1}
I20260810 03:30:45.505281 22196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:30:45.504415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:34901 (local address 127.27.161.1:43001)
0810 03:30:45.504978 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0810 03:30:45.505085 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.62:34901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":313,"threads_started":1}
I20260810 03:30:45.969009 22200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:30:45.968242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:34901 (local address 127.27.161.2:33793)
0810 03:30:45.968753 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0810 03:30:45.968837 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.62:34901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":243,"threads_started":1}
I20260810 03:30:46.445655 22201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:30:46.444821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:34901 (local address 127.27.161.3:42085)
0810 03:30:46.445330 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0810 03:30:46.445422 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.62:34901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":246,"threads_started":1}
I20260810 03:30:46.507192 22203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:30:46.506286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:34901 (local address 127.27.161.1:60361)
0810 03:30:46.506868 (+ 582us) negotiation.cc:107] Waiting for socket to connect
0810 03:30:46.506987 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.62:34901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":289,"threads_started":1}
I20260810 03:30:46.970978 22204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:30:46.970153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:34901 (local address 127.27.161.2:52853)
0810 03:30:46.970683 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0810 03:30:46.970788 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.62:34901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":256,"threads_started":1}
I20260810 03:30:47.204334 22205 raft_consensus.cc:482] T 2aca4f1321bd4bf0b5f757e209b5b17b P fac5507619954c158ebdd47a7d73fdc6 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 03:30:47.204559 22205 raft_consensus.cc:504] T 2aca4f1321bd4bf0b5f757e209b5b17b P fac5507619954c158ebdd47a7d73fdc6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f01667cb33c543839000d6eb9fb758d4" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 36973 } } peers { permanent_uuid: "fac5507619954c158ebdd47a7d73fdc6" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 38339 } } peers { permanent_uuid: "c9a9fd5ce6b04fd5b5d2480a800f403c" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 44627 } }
I20260810 03:30:47.205266 22205 leader_election.cc:288] T 2aca4f1321bd4bf0b5f757e209b5b17b P fac5507619954c158ebdd47a7d73fdc6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f01667cb33c543839000d6eb9fb758d4 (127.27.161.2:36973), c9a9fd5ce6b04fd5b5d2480a800f403c (127.27.161.3:44627)
W20260810 03:30:47.206468 21546 leader_election.cc:341] T 2aca4f1321bd4bf0b5f757e209b5b17b P fac5507619954c158ebdd47a7d73fdc6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f01667cb33c543839000d6eb9fb758d4 (127.27.161.2:36973): Not found: Tablet not found: 2aca4f1321bd4bf0b5f757e209b5b17b
W20260810 03:30:47.206650 21539 leader_election.cc:341] T 2aca4f1321bd4bf0b5f757e209b5b17b P fac5507619954c158ebdd47a7d73fdc6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c9a9fd5ce6b04fd5b5d2480a800f403c (127.27.161.3:44627): Not found: Tablet not found: 2aca4f1321bd4bf0b5f757e209b5b17b
I20260810 03:30:47.206796 21539 leader_election.cc:302] T 2aca4f1321bd4bf0b5f757e209b5b17b P fac5507619954c158ebdd47a7d73fdc6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fac5507619954c158ebdd47a7d73fdc6; no voters: c9a9fd5ce6b04fd5b5d2480a800f403c, f01667cb33c543839000d6eb9fb758d4
I20260810 03:30:47.207057 22205 raft_consensus.cc:2726] T 2aca4f1321bd4bf0b5f757e209b5b17b P fac5507619954c158ebdd47a7d73fdc6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 03:30:47.447574 22206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:30:47.446734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:34901 (local address 127.27.161.3:44741)
0810 03:30:47.447276 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0810 03:30:47.447368 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.62:34901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":281,"threads_started":1}
I20260810 03:30:47.509012 22210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:30:47.508172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:34901 (local address 127.27.161.1:47649)
0810 03:30:47.508663 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0810 03:30:47.508795 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.62:34901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":221,"threads_started":1}
I20260810 03:30:47.972910 22212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:30:47.972026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:34901 (local address 127.27.161.2:60539)
0810 03:30:47.972644 (+ 618us) negotiation.cc:107] Waiting for socket to connect
0810 03:30:47.972742 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.62:34901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":308,"threads_started":1}
I20260810 03:30:48.449419 22215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:30:48.448560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:34901 (local address 127.27.161.3:54639)
0810 03:30:48.449108 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0810 03:30:48.449229 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.62:34901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":277,"threads_started":1}
I20260810 03:30:48.510659 22216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:30:48.509889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:34901 (local address 127.27.161.1:53661)
0810 03:30:48.510368 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0810 03:30:48.510451 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.62:34901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":263,"threads_started":1}
I20260810 03:30:48.974550 22217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:30:48.973768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:34901 (local address 127.27.161.2:50593)
0810 03:30:48.974259 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0810 03:30:48.974342 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.62:34901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":263,"threads_started":1}
I20260810 03:30:49.451050 22218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:30:49.450295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:34901 (local address 127.27.161.3:33439)
0810 03:30:49.450752 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0810 03:30:49.450844 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.62:34901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":245,"threads_started":1}
I20260810 03:30:49.512540 22221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:30:49.511806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:34901 (local address 127.27.161.1:54839)
0810 03:30:49.512265 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0810 03:30:49.512360 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.62:34901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":224,"threads_started":1}
I20260810 03:30:49.976549 22223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:30:49.975755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:34901 (local address 127.27.161.2:56777)
0810 03:30:49.976266 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0810 03:30:49.976382 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.62:34901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":245,"threads_started":1}
I20260810 03:30:50.437233 28292 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iBH5/build/asan/bin/kudu pid 21451
I20260810 03:30:50.453039 22229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:30:50.452226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:34901 (local address 127.27.161.3:48073)
0810 03:30:50.452751 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0810 03:30:50.452852 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.62:34901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":274,"threads_started":1}
I20260810 03:30:50.456877 22225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:30:50.438495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39746 (local address 127.27.161.1:38339)
0810 03:30:50.438993 (+ 498us) server_negotiation.cc:206] Beginning negotiation
0810 03:30:50.439000 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:30:50.439030 (+ 30us) server_negotiation.cc:407] Connection header received
0810 03:30:50.439113 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:30:50.439120 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:30:50.439330 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:30:50.439470 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:30:50.441088 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:30:50.442118 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:30:50.443423 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:30:50.443981 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:30:50.444092 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:30:50.451218 (+ 7126us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:30:50.451232 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:30:50.451247 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:30:50.451283 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:30:50.454300 (+ 3017us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:30:50.454965 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:30:50.454971 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:30:50.454973 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:30:50.455118 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:30:50.455460 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:30:50.455463 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:30:50.455464 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:30:50.455834 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:30:50.456077 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0810 03:30:50.456352 (+ 275us) server_negotiation.cc:299] Negotiation successful
0810 03:30:50.456559 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":350,"thread_start_us":229,"threads_started":1}
I20260810 03:30:50.458232 21706 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:30:50.547770 28292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iBH5/build/asan/bin/kudu with pid 21451
I20260810 03:30:50.575982 28292 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iBH5/build/asan/bin/kudu pid 21728
I20260810 03:30:50.590160 22235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:30:50.576851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57530 (local address 127.27.161.2:36973)
0810 03:30:50.577352 (+ 501us) server_negotiation.cc:206] Beginning negotiation
0810 03:30:50.577357 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:30:50.577373 (+ 16us) server_negotiation.cc:407] Connection header received
0810 03:30:50.577430 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:30:50.577433 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:30:50.577492 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:30:50.577596 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:30:50.579082 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:30:50.579897 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:30:50.581162 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:30:50.581628 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:30:50.581743 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:30:50.584598 (+ 2855us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:30:50.584605 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:30:50.584608 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:30:50.584647 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:30:50.587280 (+ 2633us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:30:50.588015 (+ 735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:30:50.588019 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:30:50.588021 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:30:50.588171 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:30:50.588558 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:30:50.588563 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:30:50.588565 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:30:50.588885 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:30:50.589162 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0810 03:30:50.589477 (+ 315us) server_negotiation.cc:299] Negotiation successful
0810 03:30:50.589723 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":398,"thread_start_us":270,"threads_started":1}
I20260810 03:30:50.590996 21920 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:30:50.675249 28292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iBH5/build/asan/bin/kudu with pid 21728
I20260810 03:30:50.699612 28292 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iBH5/build/asan/bin/kudu pid 21935
I20260810 03:30:50.712787 22237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:30:50.700535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38218 (local address 127.27.161.3:44627)
0810 03:30:50.701122 (+ 587us) server_negotiation.cc:206] Beginning negotiation
0810 03:30:50.701126 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:30:50.701145 (+ 19us) server_negotiation.cc:407] Connection header received
0810 03:30:50.701207 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:30:50.701211 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:30:50.701297 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:30:50.701394 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:30:50.702841 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:30:50.703639 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:30:50.704801 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:30:50.705251 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:30:50.705346 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:30:50.707935 (+ 2589us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:30:50.707941 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:30:50.707944 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:30:50.707978 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:30:50.710647 (+ 2669us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:30:50.711211 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:30:50.711214 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:30:50.711216 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:30:50.711353 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:30:50.711656 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:30:50.711659 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:30:50.711661 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:30:50.711909 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:30:50.712113 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0810 03:30:50.712293 (+ 180us) server_negotiation.cc:299] Negotiation successful
0810 03:30:50.712458 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":460,"thread_start_us":311,"threads_started":1}
I20260810 03:30:50.713483 22128 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:30:50.797603 28292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iBH5/build/asan/bin/kudu with pid 21935
2026-08-10T03:30:50Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (65163 ms)
[----------] 30 tests from SecurityITest (314367 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786332336319544-28292-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 03:30:50 dist-test-slave-vpvm krb5kdc[22243](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 03:30:50 dist-test-slave-vpvm krb5kdc[22243](info): set up 2 sockets
Aug 10 03:30:50 dist-test-slave-vpvm krb5kdc[22243](info): commencing operation
krb5kdc: starting...
W20260810 03:30:52.036226 28292 mini_kdc.cc:121] Time spent starting KDC: real 1.191s user 0.000s sys 0.049s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786332336319544-28292-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786332336319544-28292-0/krb5kdc/test-user.keytab.
Aug 10 03:30:52 dist-test-slave-vpvm krb5kdc[22243](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332652, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T03:30:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T03:30:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.161.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.161.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.161.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.62.keytab.
Entry for principal kudu/127.27.161.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.62.keytab.
I20260810 03:30:52.353783 28292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iBH5/build/asan/bin/kudu
/tmp/dist-test-task24iBH5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786332336319544-28292-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786332336319544-28292-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786332336319544-28292-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786332336319544-28292-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.161.62:45277
--webserver_interface=127.27.161.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.62.keytab
--principal=kudu/127.27.161.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.161.20:33167
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.161.62:45277
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786332336319544-28292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786332336319544-28292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786332336319544-28292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:30:52.698931 22264 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:30:52.699246 22264 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:30:52.699378 22264 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:30:52.706760 22264 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:30:52.706883 22264 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:30:52.706959 22264 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:30:52.707033 22264 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:30:52.707090 22264 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:30:52.717010 22264 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.161.20:33167
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786332336319544-28292-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786332336319544-28292-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.161.62:45277
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.62.keytab
--principal=kudu/127.27.161.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.161.62:45277
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786332336319544-28292-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.161.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786332336319544-28292-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:30:52.718402 22264 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:30:52.719799 22264 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:30:52.732143 22276 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:30:52.732143 22273 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:30:52.732409 22264 server_base.cc:1034] running on GCE node
W20260810 03:30:52.732625 22271 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:30:52.733641 22264 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:30:52.735189 22264 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:30:52.736402 22264 hybrid_clock.cc:648] HybridClock initialized: now 1786332652736348 us; error 38 us; skew 500 ppm
I20260810 03:30:52.736764 22264 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:30:52 dist-test-slave-vpvm krb5kdc[22243](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332652, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.161.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:30:52.739959 22264 init.cc:377] Logged in from keytab as kudu/127.27.161.62@KRBTEST.COM (short username kudu)
I20260810 03:30:52.741503 22264 webserver.cc:460] Webserver started at http://127.27.161.62:37867/ using document root <none> and password file <none>
I20260810 03:30:52.742285 22264 fs_manager.cc:359] Metadata directory not provided
I20260810 03:30:52.742400 22264 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:30:52.742723 22264 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:30:52.745898 22264 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786332336319544-28292-0/minicluster-data/master-0/data/instance:
uuid: "2aa7e5a0a00e431bb6a609dcab5c49d4"
format_stamp: "Formatted at 2026-08-10 03:30:52 on dist-test-slave-vpvm"
server_key: "653a9d18bb74e221a648c6d04ba0747b"
server_key_iv: "9a9ea7604355d3825699cace4b80e451"
server_key_version: "encryptionkey@0"
I20260810 03:30:52.746759 22264 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786332336319544-28292-0/minicluster-data/master-0/wal/instance:
uuid: "2aa7e5a0a00e431bb6a609dcab5c49d4"
format_stamp: "Formatted at 2026-08-10 03:30:52 on dist-test-slave-vpvm"
server_key: "653a9d18bb74e221a648c6d04ba0747b"
server_key_iv: "9a9ea7604355d3825699cace4b80e451"
server_key_version: "encryptionkey@0"
I20260810 03:30:52.752373 22264 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260810 03:30:52.756189 22288 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:30:52.757587 22264 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 03:30:52.757777 22264 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786332336319544-28292-0/minicluster-data/master-0/data,/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786332336319544-28292-0/minicluster-data/master-0/wal
uuid: "2aa7e5a0a00e431bb6a609dcab5c49d4"
format_stamp: "Formatted at 2026-08-10 03:30:52 on dist-test-slave-vpvm"
server_key: "653a9d18bb74e221a648c6d04ba0747b"
server_key_iv: "9a9ea7604355d3825699cace4b80e451"
server_key_version: "encryptionkey@0"
I20260810 03:30:52.758002 22264 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786332336319544-28292-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786332336319544-28292-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786332336319544-28292-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:30:52.777777 22264 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:30:52.781774 22264 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:30:52.782075 22264 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:30:52.800239 22264 rpc_server.cc:307] RPC server started. Bound to: 127.27.161.62:45277
I20260810 03:30:52.800262 22363 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.161.62:45277 every 8 connection(s)
I20260810 03:30:52.801672 22264 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786332336319544-28292-0/minicluster-data/master-0/data/info.pb
I20260810 03:30:52.804924 28292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iBH5/build/asan/bin/kudu as pid 22264
I20260810 03:30:52.805140 28292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786332336319544-28292-0/minicluster-data/master-0/wal/instance
I20260810 03:30:52.805676 22365 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:30:52.805830 28292 external_mini_cluster.cc:1428] Setting key 4f10b732915ec80b8c62ecfa618a5e51
I20260810 03:30:52.820098 22365 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2aa7e5a0a00e431bb6a609dcab5c49d4: Bootstrap starting.
Aug 10 03:30:52 dist-test-slave-vpvm krb5kdc[22243](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332652, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.161.62@KRBTEST.COM
I20260810 03:30:52.823963 22365 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2aa7e5a0a00e431bb6a609dcab5c49d4: Neither blocks nor log segments found. Creating new log.
I20260810 03:30:52.825377 22365 log.cc:824] T 00000000000000000000000000000000 P 2aa7e5a0a00e431bb6a609dcab5c49d4: Log is configured to *not* fsync() on all Append() calls
I20260810 03:30:52.828753 22365 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2aa7e5a0a00e431bb6a609dcab5c49d4: No bootstrap required, opened a new log
I20260810 03:30:52.832037 22369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:30:52.807965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44998 (local address 127.27.161.62:45277)
0810 03:30:52.808915 (+ 950us) server_negotiation.cc:206] Beginning negotiation
0810 03:30:52.808932 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0810 03:30:52.809294 (+ 362us) server_negotiation.cc:407] Connection header received
0810 03:30:52.812068 (+ 2774us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:30:52.812099 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:30:52.812606 (+ 507us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:30:52.813077 (+ 471us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:30:52.814357 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:30:52.816336 (+ 1979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:30:52.817711 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:30:52.818795 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:30:52.818980 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:30:52.823602 (+ 4622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:30:52.823615 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:30:52.823637 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:30:52.823694 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:30:52.827275 (+ 3581us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:30:52.828040 (+ 765us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:30:52.828049 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:30:52.828057 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:30:52.828350 (+ 293us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:30:52.828690 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:30:52.828694 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:30:52.828697 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:30:52.829487 (+ 790us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:30:52.829795 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0810 03:30:52.830169 (+ 374us) server_negotiation.cc:299] Negotiation successful
0810 03:30:52.830613 (+ 444us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":671,"thread_start_us":410,"threads_started":1}
I20260810 03:30:52.837376 22365 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2aa7e5a0a00e431bb6a609dcab5c49d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2aa7e5a0a00e431bb6a609dcab5c49d4" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 45277 } }
I20260810 03:30:52.837754 22365 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2aa7e5a0a00e431bb6a609dcab5c49d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:30:52.837829 22365 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2aa7e5a0a00e431bb6a609dcab5c49d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2aa7e5a0a00e431bb6a609dcab5c49d4, State: Initialized, Role: FOLLOWER
I20260810 03:30:52.838410 22365 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2aa7e5a0a00e431bb6a609dcab5c49d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2aa7e5a0a00e431bb6a609dcab5c49d4" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 45277 } }
I20260810 03:30:52.838632 22365 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2aa7e5a0a00e431bb6a609dcab5c49d4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 03:30:52.838734 22365 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2aa7e5a0a00e431bb6a609dcab5c49d4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 03:30:52.838860 22365 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2aa7e5a0a00e431bb6a609dcab5c49d4 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:30:52.841027 22365 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2aa7e5a0a00e431bb6a609dcab5c49d4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2aa7e5a0a00e431bb6a609dcab5c49d4" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 45277 } }
I20260810 03:30:52.841542 22365 leader_election.cc:302] T 00000000000000000000000000000000 P 2aa7e5a0a00e431bb6a609dcab5c49d4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2aa7e5a0a00e431bb6a609dcab5c49d4; no voters:
I20260810 03:30:52.842159 22365 leader_election.cc:288] T 00000000000000000000000000000000 P 2aa7e5a0a00e431bb6a609dcab5c49d4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 03:30:52.842295 22371 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2aa7e5a0a00e431bb6a609dcab5c49d4 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:30:52.842897 22371 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2aa7e5a0a00e431bb6a609dcab5c49d4 [term 1 LEADER]: Becoming Leader. State: Replica: 2aa7e5a0a00e431bb6a609dcab5c49d4, State: Running, Role: LEADER
I20260810 03:30:52.843451 22371 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2aa7e5a0a00e431bb6a609dcab5c49d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2aa7e5a0a00e431bb6a609dcab5c49d4" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 45277 } }
I20260810 03:30:52.844203 22365 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2aa7e5a0a00e431bb6a609dcab5c49d4 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:30:52.846247 22373 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2aa7e5a0a00e431bb6a609dcab5c49d4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2aa7e5a0a00e431bb6a609dcab5c49d4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2aa7e5a0a00e431bb6a609dcab5c49d4" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 45277 } } }
I20260810 03:30:52.846487 22376 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2aa7e5a0a00e431bb6a609dcab5c49d4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2aa7e5a0a00e431bb6a609dcab5c49d4. Latest consensus state: current_term: 1 leader_uuid: "2aa7e5a0a00e431bb6a609dcab5c49d4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2aa7e5a0a00e431bb6a609dcab5c49d4" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 45277 } } }
I20260810 03:30:52.846758 22373 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2aa7e5a0a00e431bb6a609dcab5c49d4 [sys.catalog]: This master's current role is: LEADER
I20260810 03:30:52.846838 22376 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2aa7e5a0a00e431bb6a609dcab5c49d4 [sys.catalog]: This master's current role is: LEADER
I20260810 03:30:52.848202 22385 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 03:30:52.853576 22385 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 03:30:52.863034 22385 catalog_manager.cc:1347] Generated new cluster ID: 3e80c33f4abc4e70b804372dfe55a88a
I20260810 03:30:52.863165 22385 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 03:30:52.898905 22385 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 03:30:52.900902 22385 catalog_manager.cc:1504] Loading token signing keys...
I20260810 03:30:52.912158 22385 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2aa7e5a0a00e431bb6a609dcab5c49d4: Generated new TSK 0
I20260810 03:30:52.913254 22385 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.161.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.161.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.161.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.1.keytab.
Entry for principal kudu/127.27.161.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.1.keytab.
I20260810 03:30:52.992707 28292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iBH5/build/asan/bin/kudu
/tmp/dist-test-task24iBH5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786332336319544-28292-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786332336319544-28292-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786332336319544-28292-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786332336319544-28292-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.161.1:0
--local_ip_for_outbound_sockets=127.27.161.1
--webserver_interface=127.27.161.1
--webserver_port=0
--tserver_master_addrs=127.27.161.62:45277
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.1.keytab
--principal=kudu/127.27.161.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.161.20:33167
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786332336319544-28292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786332336319544-28292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786332336319544-28292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:30:53.351326 22406 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:30:53.351634 22406 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:30:53.351742 22406 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:30:53.359284 22406 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:30:53.359421 22406 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:30:53.359548 22406 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.161.1
I20260810 03:30:53.370745 22406 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.161.20:33167
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786332336319544-28292-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786332336319544-28292-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.1.keytab
--principal=kudu/127.27.161.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.161.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786332336319544-28292-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.161.1
--webserver_port=0
--tserver_master_addrs=127.27.161.62:45277
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.161.1
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786332336319544-28292-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:30:53.372119 22406 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:30:53.373584 22406 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:30:53.386984 22419 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:30:53.387012 22415 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:30:53.387013 22416 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:30:53.388028 22406 server_base.cc:1034] running on GCE node
I20260810 03:30:53.388651 22406 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:30:53.389634 22406 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:30:53.390847 22406 hybrid_clock.cc:648] HybridClock initialized: now 1786332653390793 us; error 40 us; skew 500 ppm
I20260810 03:30:53.391183 22406 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:30:53 dist-test-slave-vpvm krb5kdc[22243](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332653, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.161.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:30:53.394428 22406 init.cc:377] Logged in from keytab as kudu/127.27.161.1@KRBTEST.COM (short username kudu)
I20260810 03:30:53.395984 22406 webserver.cc:460] Webserver started at http://127.27.161.1:36759/ using document root <none> and password file <none>
I20260810 03:30:53.396719 22406 fs_manager.cc:359] Metadata directory not provided
I20260810 03:30:53.396847 22406 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:30:53.397173 22406 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:30:53.400341 22406 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786332336319544-28292-0/minicluster-data/ts-0/data/instance:
uuid: "8877b25312134dcc8ad309ef4c17f84e"
format_stamp: "Formatted at 2026-08-10 03:30:53 on dist-test-slave-vpvm"
server_key: "8ded74afc354415e795dd5a6810a2e13"
server_key_iv: "b1aed625375e5816b9ea3a5a929cd275"
server_key_version: "encryptionkey@0"
I20260810 03:30:53.401125 22406 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786332336319544-28292-0/minicluster-data/ts-0/wal/instance:
uuid: "8877b25312134dcc8ad309ef4c17f84e"
format_stamp: "Formatted at 2026-08-10 03:30:53 on dist-test-slave-vpvm"
server_key: "8ded74afc354415e795dd5a6810a2e13"
server_key_iv: "b1aed625375e5816b9ea3a5a929cd275"
server_key_version: "encryptionkey@0"
I20260810 03:30:53.406605 22406 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260810 03:30:53.410303 22434 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:30:53.411813 22406 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 03:30:53.412017 22406 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786332336319544-28292-0/minicluster-data/ts-0/data,/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786332336319544-28292-0/minicluster-data/ts-0/wal
uuid: "8877b25312134dcc8ad309ef4c17f84e"
format_stamp: "Formatted at 2026-08-10 03:30:53 on dist-test-slave-vpvm"
server_key: "8ded74afc354415e795dd5a6810a2e13"
server_key_iv: "b1aed625375e5816b9ea3a5a929cd275"
server_key_version: "encryptionkey@0"
I20260810 03:30:53.412213 22406 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786332336319544-28292-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786332336319544-28292-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786332336319544-28292-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:30:53.428354 22406 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:30:53.432482 22406 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:30:53.432739 22406 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:30:53.433720 22406 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:30:53.435214 22406 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:30:53.435290 22406 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:30:53.435390 22406 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:30:53.435433 22406 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:30:53.467025 22406 rpc_server.cc:307] RPC server started. Bound to: 127.27.161.1:33847
I20260810 03:30:53.467049 22622 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.161.1:33847 every 8 connection(s)
I20260810 03:30:53.468315 22406 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786332336319544-28292-0/minicluster-data/ts-0/data/info.pb
I20260810 03:30:53.475814 28292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iBH5/build/asan/bin/kudu as pid 22406
I20260810 03:30:53.476053 28292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786332336319544-28292-0/minicluster-data/ts-0/wal/instance
I20260810 03:30:53.476882 28292 external_mini_cluster.cc:1428] Setting key a7c75e85e97e6b745377ff8cab200439
Aug 10 03:30:53 dist-test-slave-vpvm krb5kdc[22243](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332653, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.161.1@KRBTEST.COM for kudu/127.27.161.62@KRBTEST.COM
I20260810 03:30:53.489118 22626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:30:53.470901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.161.1:58741 (local address 127.27.161.62:45277)
0810 03:30:53.471366 (+ 465us) server_negotiation.cc:206] Beginning negotiation
0810 03:30:53.471371 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:30:53.472341 (+ 970us) server_negotiation.cc:407] Connection header received
0810 03:30:53.474238 (+ 1897us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:30:53.474242 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:30:53.474313 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:30:53.474415 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:30:53.475960 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:30:53.477085 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:30:53.478832 (+ 1747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:30:53.479400 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:30:53.479540 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:30:53.482991 (+ 3451us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:30:53.482998 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:30:53.483002 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:30:53.483044 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:30:53.485937 (+ 2893us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:30:53.486717 (+ 780us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:30:53.486722 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:30:53.486725 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:30:53.486874 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:30:53.487308 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:30:53.487312 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:30:53.487315 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:30:53.487663 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:30:53.487820 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0810 03:30:53.488538 (+ 718us) server_negotiation.cc:299] Negotiation successful
0810 03:30:53.488789 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":219,"threads_started":1}
I20260810 03:30:53.490056 22627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:30:53.471461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:45277 (local address 127.27.161.1:58741)
0810 03:30:53.472183 (+ 722us) negotiation.cc:107] Waiting for socket to connect
0810 03:30:53.472221 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0810 03:30:53.473937 (+ 1716us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:30:53.474626 (+ 689us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:30:53.474633 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:30:53.475147 (+ 514us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:30:53.475767 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:30:53.475785 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:30:53.477240 (+ 1455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:30:53.477244 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:30:53.478696 (+ 1452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:30:53.478712 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:30:53.479631 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:30:53.479637 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:30:53.479851 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:30:53.480487 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:30:53.480511 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:30:53.482741 (+ 2230us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:30:53.486106 (+ 3365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:30:53.486114 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:30:53.486128 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:30:53.486570 (+ 442us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:30:53.486972 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:30:53.486975 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:30:53.486977 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:30:53.487191 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:30:53.487791 (+ 600us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:30:53.487798 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:30:53.488147 (+ 349us) client_negotiation.cc:769] Sending connection context
0810 03:30:53.488381 (+ 234us) client_negotiation.cc:240] Negotiation successful
0810 03:30:53.488819 (+ 438us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":487,"thread_start_us":277,"threads_started":1}
I20260810 03:30:53.491686 22623 heartbeater.cc:347] Connected to a master server at 127.27.161.62:45277
I20260810 03:30:53.491978 22623 heartbeater.cc:464] Registering TS with master...
I20260810 03:30:53.492731 22623 heartbeater.cc:511] Master 127.27.161.62:45277 requested a full tablet report, sending...
I20260810 03:30:53.494889 22315 ts_manager.cc:194] Registered new tserver with Master: 8877b25312134dcc8ad309ef4c17f84e (127.27.161.1:33847)
I20260810 03:30:53.496359 22315 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.161.1@KRBTEST.COM'} at 127.27.161.1:58741
WARNING: no policy specified for kudu/127.27.161.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.161.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.161.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.2.keytab.
Entry for principal kudu/127.27.161.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.2.keytab.
I20260810 03:30:53.547276 28292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iBH5/build/asan/bin/kudu
/tmp/dist-test-task24iBH5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786332336319544-28292-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786332336319544-28292-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786332336319544-28292-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786332336319544-28292-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.161.2:0
--local_ip_for_outbound_sockets=127.27.161.2
--webserver_interface=127.27.161.2
--webserver_port=0
--tserver_master_addrs=127.27.161.62:45277
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.2.keytab
--principal=kudu/127.27.161.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.161.20:33167
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786332336319544-28292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786332336319544-28292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786332336319544-28292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:30:53.907371 22631 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:30:53.907725 22631 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:30:53.907866 22631 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:30:53.915802 22631 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:30:53.915939 22631 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:30:53.916121 22631 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.161.2
I20260810 03:30:53.926579 22631 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.161.20:33167
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786332336319544-28292-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786332336319544-28292-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.2.keytab
--principal=kudu/127.27.161.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.161.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786332336319544-28292-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.161.2
--webserver_port=0
--tserver_master_addrs=127.27.161.62:45277
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.161.2
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786332336319544-28292-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:30:53.928016 22631 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:30:53.929499 22631 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:30:53.943295 22644 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:30:53.943364 22642 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:30:53.943324 22647 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:30:53.944403 22631 server_base.cc:1034] running on GCE node
I20260810 03:30:53.945058 22631 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:30:53.946171 22631 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:30:53.947393 22631 hybrid_clock.cc:648] HybridClock initialized: now 1786332653947336 us; error 45 us; skew 500 ppm
I20260810 03:30:53.947721 22631 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:30:53 dist-test-slave-vpvm krb5kdc[22243](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332653, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.161.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:30:53.951119 22631 init.cc:377] Logged in from keytab as kudu/127.27.161.2@KRBTEST.COM (short username kudu)
I20260810 03:30:53.952736 22631 webserver.cc:460] Webserver started at http://127.27.161.2:44787/ using document root <none> and password file <none>
I20260810 03:30:53.953465 22631 fs_manager.cc:359] Metadata directory not provided
I20260810 03:30:53.953593 22631 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:30:53.953912 22631 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:30:53.957067 22631 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786332336319544-28292-0/minicluster-data/ts-1/data/instance:
uuid: "6a2f5dbb3a4a46ef9050b93425e48df9"
format_stamp: "Formatted at 2026-08-10 03:30:53 on dist-test-slave-vpvm"
server_key: "3b0719cf00fc2e88943e5a245209d064"
server_key_iv: "eeffd0854821ee06aa7c78c892ba385f"
server_key_version: "encryptionkey@0"
I20260810 03:30:53.957864 22631 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786332336319544-28292-0/minicluster-data/ts-1/wal/instance:
uuid: "6a2f5dbb3a4a46ef9050b93425e48df9"
format_stamp: "Formatted at 2026-08-10 03:30:53 on dist-test-slave-vpvm"
server_key: "3b0719cf00fc2e88943e5a245209d064"
server_key_iv: "eeffd0854821ee06aa7c78c892ba385f"
server_key_version: "encryptionkey@0"
I20260810 03:30:53.963471 22631 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260810 03:30:53.967154 22656 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:30:53.968518 22631 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 03:30:53.968693 22631 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786332336319544-28292-0/minicluster-data/ts-1/data,/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786332336319544-28292-0/minicluster-data/ts-1/wal
uuid: "6a2f5dbb3a4a46ef9050b93425e48df9"
format_stamp: "Formatted at 2026-08-10 03:30:53 on dist-test-slave-vpvm"
server_key: "3b0719cf00fc2e88943e5a245209d064"
server_key_iv: "eeffd0854821ee06aa7c78c892ba385f"
server_key_version: "encryptionkey@0"
I20260810 03:30:53.968896 22631 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786332336319544-28292-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786332336319544-28292-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786332336319544-28292-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:30:53.994833 22631 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:30:53.999428 22631 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:30:53.999734 22631 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:30:54.000797 22631 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:30:54.002312 22631 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:30:54.002398 22631 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:30:54.002522 22631 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:30:54.002586 22631 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:30:54.036513 22631 rpc_server.cc:307] RPC server started. Bound to: 127.27.161.2:40219
I20260810 03:30:54.036583 22823 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.161.2:40219 every 8 connection(s)
I20260810 03:30:54.037894 22631 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786332336319544-28292-0/minicluster-data/ts-1/data/info.pb
I20260810 03:30:54.041482 28292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iBH5/build/asan/bin/kudu as pid 22631
I20260810 03:30:54.041692 28292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786332336319544-28292-0/minicluster-data/ts-1/wal/instance
I20260810 03:30:54.042273 28292 external_mini_cluster.cc:1428] Setting key 112d33e52ad604a2be14700e7823fa4e
Aug 10 03:30:54 dist-test-slave-vpvm krb5kdc[22243](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332653, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.161.2@KRBTEST.COM for kudu/127.27.161.62@KRBTEST.COM
I20260810 03:30:54.059717 22828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:30:54.040425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.161.2:60115 (local address 127.27.161.62:45277)
0810 03:30:54.040933 (+ 508us) server_negotiation.cc:206] Beginning negotiation
0810 03:30:54.040940 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:30:54.042002 (+ 1062us) server_negotiation.cc:407] Connection header received
0810 03:30:54.044163 (+ 2161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:30:54.044167 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:30:54.044231 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:30:54.044334 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:30:54.046012 (+ 1678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:30:54.047082 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:30:54.048691 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:30:54.049276 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:30:54.049399 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:30:54.053166 (+ 3767us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:30:54.053173 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:30:54.053177 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:30:54.053221 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:30:54.056388 (+ 3167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:30:54.057238 (+ 850us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:30:54.057243 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:30:54.057246 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:30:54.057413 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:30:54.057901 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:30:54.057905 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:30:54.057908 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:30:54.058263 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:30:54.058406 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0810 03:30:54.059110 (+ 704us) server_negotiation.cc:299] Negotiation successful
0810 03:30:54.059336 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":227,"threads_started":1}
I20260810 03:30:54.060861 22830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:30:54.040981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:45277 (local address 127.27.161.2:60115)
0810 03:30:54.041836 (+ 855us) negotiation.cc:107] Waiting for socket to connect
0810 03:30:54.041876 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0810 03:30:54.043862 (+ 1986us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:30:54.044607 (+ 745us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:30:54.044616 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:30:54.045174 (+ 558us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:30:54.045830 (+ 656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:30:54.045848 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:30:54.047266 (+ 1418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:30:54.047270 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:30:54.048532 (+ 1262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:30:54.048547 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:30:54.049523 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:30:54.049529 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:30:54.049743 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:30:54.050506 (+ 763us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:30:54.050531 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:30:54.052888 (+ 2357us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:30:54.056576 (+ 3688us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:30:54.056588 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:30:54.056604 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:30:54.057107 (+ 503us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:30:54.057512 (+ 405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:30:54.057516 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:30:54.057518 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:30:54.057791 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:30:54.058392 (+ 601us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:30:54.058399 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:30:54.058756 (+ 357us) client_negotiation.cc:769] Sending connection context
0810 03:30:54.059006 (+ 250us) client_negotiation.cc:240] Negotiation successful
0810 03:30:54.059439 (+ 433us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":561,"thread_start_us":336,"threads_started":1}
I20260810 03:30:54.062531 22824 heartbeater.cc:347] Connected to a master server at 127.27.161.62:45277
I20260810 03:30:54.062876 22824 heartbeater.cc:464] Registering TS with master...
I20260810 03:30:54.063634 22824 heartbeater.cc:511] Master 127.27.161.62:45277 requested a full tablet report, sending...
I20260810 03:30:54.065223 22315 ts_manager.cc:194] Registered new tserver with Master: 6a2f5dbb3a4a46ef9050b93425e48df9 (127.27.161.2:40219)
I20260810 03:30:54.066368 22315 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.161.2@KRBTEST.COM'} at 127.27.161.2:60115
WARNING: no policy specified for kudu/127.27.161.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.161.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.161.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.3.keytab.
Entry for principal kudu/127.27.161.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.3.keytab.
I20260810 03:30:54.112809 28292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iBH5/build/asan/bin/kudu
/tmp/dist-test-task24iBH5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786332336319544-28292-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786332336319544-28292-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786332336319544-28292-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786332336319544-28292-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.161.3:0
--local_ip_for_outbound_sockets=127.27.161.3
--webserver_interface=127.27.161.3
--webserver_port=0
--tserver_master_addrs=127.27.161.62:45277
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.3.keytab
--principal=kudu/127.27.161.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.161.20:33167
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786332336319544-28292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786332336319544-28292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786332336319544-28292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:30:54.470095 22833 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:30:54.470443 22833 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:30:54.470594 22833 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:30:54.478257 22833 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:30:54.478382 22833 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:30:54.478536 22833 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.161.3
I20260810 03:30:54.488652 22833 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.161.20:33167
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786332336319544-28292-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786332336319544-28292-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.3.keytab
--principal=kudu/127.27.161.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.161.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786332336319544-28292-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.161.3
--webserver_port=0
--tserver_master_addrs=127.27.161.62:45277
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.161.3
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786332336319544-28292-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:30:54.490054 22833 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:30:54.491482 22833 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 03:30:54.499553 22623 heartbeater.cc:503] Master 127.27.161.62:45277 was elected leader, sending a full tablet report...
W20260810 03:30:54.505290 22841 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:30:54.505316 22843 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:30:54.505484 22846 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:30:54.506115 22833 server_base.cc:1034] running on GCE node
I20260810 03:30:54.506779 22833 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:30:54.507823 22833 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:30:54.509020 22833 hybrid_clock.cc:648] HybridClock initialized: now 1786332654508966 us; error 68 us; skew 500 ppm
I20260810 03:30:54.509362 22833 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:30:54 dist-test-slave-vpvm krb5kdc[22243](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332654, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.161.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:30:54.512483 22833 init.cc:377] Logged in from keytab as kudu/127.27.161.3@KRBTEST.COM (short username kudu)
I20260810 03:30:54.513939 22833 webserver.cc:460] Webserver started at http://127.27.161.3:39597/ using document root <none> and password file <none>
I20260810 03:30:54.514691 22833 fs_manager.cc:359] Metadata directory not provided
I20260810 03:30:54.514814 22833 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:30:54.515136 22833 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:30:54.518337 22833 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786332336319544-28292-0/minicluster-data/ts-2/data/instance:
uuid: "0c4fd1646afb442498ee265dd832be80"
format_stamp: "Formatted at 2026-08-10 03:30:54 on dist-test-slave-vpvm"
server_key: "ea461a281c7ab31553b13831e9890d56"
server_key_iv: "5d8cbba386573d65ee12d71e9d25b81f"
server_key_version: "encryptionkey@0"
I20260810 03:30:54.519119 22833 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786332336319544-28292-0/minicluster-data/ts-2/wal/instance:
uuid: "0c4fd1646afb442498ee265dd832be80"
format_stamp: "Formatted at 2026-08-10 03:30:54 on dist-test-slave-vpvm"
server_key: "ea461a281c7ab31553b13831e9890d56"
server_key_iv: "5d8cbba386573d65ee12d71e9d25b81f"
server_key_version: "encryptionkey@0"
I20260810 03:30:54.524657 22833 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260810 03:30:54.528369 22856 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:30:54.529747 22833 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 03:30:54.529908 22833 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786332336319544-28292-0/minicluster-data/ts-2/data,/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786332336319544-28292-0/minicluster-data/ts-2/wal
uuid: "0c4fd1646afb442498ee265dd832be80"
format_stamp: "Formatted at 2026-08-10 03:30:54 on dist-test-slave-vpvm"
server_key: "ea461a281c7ab31553b13831e9890d56"
server_key_iv: "5d8cbba386573d65ee12d71e9d25b81f"
server_key_version: "encryptionkey@0"
I20260810 03:30:54.530145 22833 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786332336319544-28292-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786332336319544-28292-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786332336319544-28292-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:30:54.544929 22833 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:30:54.549213 22833 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:30:54.549535 22833 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:30:54.550637 22833 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:30:54.552060 22833 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:30:54.552152 22833 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:30:54.552289 22833 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:30:54.552349 22833 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:30:54.584448 22833 rpc_server.cc:307] RPC server started. Bound to: 127.27.161.3:36013
I20260810 03:30:54.584510 23033 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.161.3:36013 every 8 connection(s)
I20260810 03:30:54.585803 22833 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786332336319544-28292-0/minicluster-data/ts-2/data/info.pb
I20260810 03:30:54.596222 28292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iBH5/build/asan/bin/kudu as pid 22833
I20260810 03:30:54.596408 28292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786332336319544-28292-0/minicluster-data/ts-2/wal/instance
I20260810 03:30:54.596868 28292 external_mini_cluster.cc:1428] Setting key c06c30023650993f799b121bc3a3277c
Aug 10 03:30:54 dist-test-slave-vpvm krb5kdc[22243](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332654, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.161.3@KRBTEST.COM for kudu/127.27.161.62@KRBTEST.COM
I20260810 03:30:54.606377 23037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:30:54.588411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.161.3:43781 (local address 127.27.161.62:45277)
0810 03:30:54.588856 (+ 445us) server_negotiation.cc:206] Beginning negotiation
0810 03:30:54.588861 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:30:54.589844 (+ 983us) server_negotiation.cc:407] Connection header received
0810 03:30:54.591810 (+ 1966us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:30:54.591816 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:30:54.591915 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:30:54.592023 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:30:54.593636 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:30:54.594460 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:30:54.596139 (+ 1679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:30:54.596636 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:30:54.596759 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:30:54.600490 (+ 3731us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:30:54.600495 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:30:54.600499 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:30:54.600536 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:30:54.603141 (+ 2605us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:30:54.603930 (+ 789us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:30:54.603935 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:30:54.603938 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:30:54.604069 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:30:54.604524 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:30:54.604528 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:30:54.604530 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:30:54.604811 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:30:54.604957 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0810 03:30:54.605695 (+ 738us) server_negotiation.cc:299] Negotiation successful
0810 03:30:54.605941 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":337,"thread_start_us":209,"threads_started":1}
I20260810 03:30:54.607436 23039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:30:54.588885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:45277 (local address 127.27.161.3:43781)
0810 03:30:54.589665 (+ 780us) negotiation.cc:107] Waiting for socket to connect
0810 03:30:54.589705 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0810 03:30:54.591504 (+ 1799us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:30:54.592292 (+ 788us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:30:54.592300 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:30:54.592838 (+ 538us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:30:54.593450 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:30:54.593469 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:30:54.594639 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:30:54.594643 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:30:54.595970 (+ 1327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:30:54.595994 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:30:54.596892 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:30:54.596900 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:30:54.597176 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:30:54.597870 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:30:54.597905 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:30:54.600265 (+ 2360us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:30:54.603312 (+ 3047us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:30:54.603323 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:30:54.603343 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:30:54.603805 (+ 462us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:30:54.604185 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:30:54.604190 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:30:54.604194 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:30:54.604404 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:30:54.604944 (+ 540us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:30:54.604952 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:30:54.605329 (+ 377us) client_negotiation.cc:769] Sending connection context
0810 03:30:54.605589 (+ 260us) client_negotiation.cc:240] Negotiation successful
0810 03:30:54.606225 (+ 636us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":520,"thread_start_us":300,"threads_started":1}
I20260810 03:30:54.608922 23034 heartbeater.cc:347] Connected to a master server at 127.27.161.62:45277
I20260810 03:30:54.609225 23034 heartbeater.cc:464] Registering TS with master...
I20260810 03:30:54.609997 23034 heartbeater.cc:511] Master 127.27.161.62:45277 requested a full tablet report, sending...
I20260810 03:30:54.611519 22315 ts_manager.cc:194] Registered new tserver with Master: 0c4fd1646afb442498ee265dd832be80 (127.27.161.3:36013)
I20260810 03:30:54.612370 28292 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 03:30:54.612493 22315 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.161.3@KRBTEST.COM'} at 127.27.161.3:43781
I20260810 03:30:54.630055 23037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:30:54.617010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45010 (local address 127.27.161.62:45277)
0810 03:30:54.617167 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0810 03:30:54.617173 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:30:54.617622 (+ 449us) server_negotiation.cc:407] Connection header received
0810 03:30:54.617797 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:30:54.617801 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:30:54.617881 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:30:54.618007 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:30:54.619483 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:30:54.620226 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:30:54.621659 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:30:54.622118 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:30:54.622236 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:30:54.624694 (+ 2458us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:30:54.624701 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:30:54.624706 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:30:54.624741 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:30:54.627237 (+ 2496us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:30:54.627950 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:30:54.627957 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:30:54.627960 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:30:54.628110 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:30:54.628493 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:30:54.628497 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:30:54.628500 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:30:54.628856 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:30:54.629097 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0810 03:30:54.629466 (+ 369us) server_negotiation.cc:299] Negotiation successful
0810 03:30:54.629719 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260810 03:30:54.634186 22315 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45010:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 03:30:54.641336 22315 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 03:30:54.676402 23057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:30:54.653417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.3:36013 (local address 127.0.0.1:48404)
0810 03:30:54.665118 (+ 11701us) negotiation.cc:107] Waiting for socket to connect
0810 03:30:54.665199 (+ 81us) client_negotiation.cc:174] Beginning negotiation
0810 03:30:54.665425 (+ 226us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:30:54.667794 (+ 2369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:30:54.667804 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:30:54.667842 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:30:54.668327 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:30:54.668349 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:30:54.670654 (+ 2305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:30:54.670659 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:30:54.673347 (+ 2688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:30:54.673368 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:30:54.675273 (+ 1905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:30:54.675281 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:30:54.675477 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:30:54.675543 (+ 66us) client_negotiation.cc:769] Sending connection context
0810 03:30:54.675659 (+ 116us) client_negotiation.cc:240] Negotiation successful
0810 03:30:54.675912 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11349,"mutex_wait_us":1,"thread_start_us":15655,"threads_started":1}
I20260810 03:30:54.677007 23055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:30:54.653769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.2:40219 (local address 127.0.0.1:58606)
0810 03:30:54.668852 (+ 15083us) negotiation.cc:107] Waiting for socket to connect
0810 03:30:54.668874 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0810 03:30:54.669001 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:30:54.670053 (+ 1052us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:30:54.670059 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:30:54.670086 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:30:54.670474 (+ 388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:30:54.670487 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:30:54.672704 (+ 2217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:30:54.672710 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:30:54.674449 (+ 1739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:30:54.674462 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:30:54.676236 (+ 1774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:30:54.676243 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:30:54.676450 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:30:54.676483 (+ 33us) client_negotiation.cc:769] Sending connection context
0810 03:30:54.676554 (+ 71us) client_negotiation.cc:240] Negotiation successful
0810 03:30:54.676665 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14939,"thread_start_us":11870,"threads_started":1}
I20260810 03:30:54.677160 23056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:30:54.653657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48404 (local address 127.27.161.3:36013)
0810 03:30:54.666962 (+ 13305us) server_negotiation.cc:206] Beginning negotiation
0810 03:30:54.666972 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 03:30:54.667012 (+ 40us) server_negotiation.cc:407] Connection header received
0810 03:30:54.667125 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:30:54.667133 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:30:54.667428 (+ 295us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:30:54.668580 (+ 1152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:30:54.668890 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:30:54.670497 (+ 1607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:30:54.673533 (+ 3036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:30:54.675057 (+ 1524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:30:54.675171 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:30:54.676015 (+ 844us) server_negotiation.cc:1035] Waiting for connection context
0810 03:30:54.676095 (+ 80us) server_negotiation.cc:299] Negotiation successful
0810 03:30:54.676275 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13100,"thread_start_us":12917,"threads_started":1}
I20260810 03:30:54.678162 23058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:30:54.653927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58606 (local address 127.27.161.2:40219)
0810 03:30:54.669194 (+ 15267us) server_negotiation.cc:206] Beginning negotiation
0810 03:30:54.669203 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 03:30:54.669233 (+ 30us) server_negotiation.cc:407] Connection header received
0810 03:30:54.669332 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:30:54.669341 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:30:54.669587 (+ 246us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:30:54.670731 (+ 1144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:30:54.670973 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:30:54.672531 (+ 1558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:30:54.674606 (+ 2075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:30:54.675968 (+ 1362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:30:54.677431 (+ 1463us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:30:54.677624 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0810 03:30:54.677689 (+ 65us) server_negotiation.cc:299] Negotiation successful
0810 03:30:54.677844 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15080,"thread_start_us":15802,"threads_started":1}
I20260810 03:30:54.680812 23060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:30:54.655454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.1:33847 (local address 127.0.0.1:51410)
0810 03:30:54.671874 (+ 16420us) negotiation.cc:107] Waiting for socket to connect
0810 03:30:54.671890 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 03:30:54.672015 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:30:54.672596 (+ 581us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:30:54.672601 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:30:54.672626 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:30:54.673181 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:30:54.673194 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:30:54.675091 (+ 1897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:30:54.675097 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:30:54.678059 (+ 2962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:30:54.678075 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:30:54.679934 (+ 1859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:30:54.679940 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:30:54.680105 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:30:54.680140 (+ 35us) client_negotiation.cc:769] Sending connection context
0810 03:30:54.680313 (+ 173us) client_negotiation.cc:240] Negotiation successful
0810 03:30:54.680444 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16225,"mutex_wait_us":919,"thread_start_us":16007,"threads_started":1}
I20260810 03:30:54.680816 23063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:30:54.656107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51410 (local address 127.27.161.1:33847)
0810 03:30:54.669793 (+ 13686us) server_negotiation.cc:206] Beginning negotiation
0810 03:30:54.669802 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 03:30:54.672023 (+ 2221us) server_negotiation.cc:407] Connection header received
0810 03:30:54.672184 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:30:54.672191 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:30:54.672404 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:30:54.672582 (+ 178us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:30:54.673366 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:30:54.674937 (+ 1571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:30:54.678319 (+ 3382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:30:54.679698 (+ 1379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:30:54.679849 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:30:54.680067 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0810 03:30:54.680313 (+ 246us) server_negotiation.cc:299] Negotiation successful
0810 03:30:54.680483 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13500,"thread_start_us":13342,"threads_started":1}
I20260810 03:30:54.681355 22723 tablet_service.cc:1467] Processing CreateTablet for tablet d15dd84e6aa34ce89588cede6d943e8d (DEFAULT_TABLE table=test-table [id=9dce2492570c4998ae56b26786508848]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:30:54.681371 22931 tablet_service.cc:1467] Processing CreateTablet for tablet d15dd84e6aa34ce89588cede6d943e8d (DEFAULT_TABLE table=test-table [id=9dce2492570c4998ae56b26786508848]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:30:54.682778 22723 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d15dd84e6aa34ce89588cede6d943e8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:30:54.682852 22931 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d15dd84e6aa34ce89588cede6d943e8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:30:54.693885 23064 tablet_bootstrap.cc:492] T d15dd84e6aa34ce89588cede6d943e8d P 0c4fd1646afb442498ee265dd832be80: Bootstrap starting.
I20260810 03:30:54.693444 22509 tablet_service.cc:1467] Processing CreateTablet for tablet d15dd84e6aa34ce89588cede6d943e8d (DEFAULT_TABLE table=test-table [id=9dce2492570c4998ae56b26786508848]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:30:54.695125 22509 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d15dd84e6aa34ce89588cede6d943e8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:30:54.696832 23064 tablet_bootstrap.cc:654] T d15dd84e6aa34ce89588cede6d943e8d P 0c4fd1646afb442498ee265dd832be80: Neither blocks nor log segments found. Creating new log.
I20260810 03:30:54.697695 23065 tablet_bootstrap.cc:492] T d15dd84e6aa34ce89588cede6d943e8d P 6a2f5dbb3a4a46ef9050b93425e48df9: Bootstrap starting.
I20260810 03:30:54.700170 23064 log.cc:824] T d15dd84e6aa34ce89588cede6d943e8d P 0c4fd1646afb442498ee265dd832be80: Log is configured to *not* fsync() on all Append() calls
I20260810 03:30:54.701243 23065 tablet_bootstrap.cc:654] T d15dd84e6aa34ce89588cede6d943e8d P 6a2f5dbb3a4a46ef9050b93425e48df9: Neither blocks nor log segments found. Creating new log.
I20260810 03:30:54.702476 23065 log.cc:824] T d15dd84e6aa34ce89588cede6d943e8d P 6a2f5dbb3a4a46ef9050b93425e48df9: Log is configured to *not* fsync() on all Append() calls
I20260810 03:30:54.703528 23064 tablet_bootstrap.cc:492] T d15dd84e6aa34ce89588cede6d943e8d P 0c4fd1646afb442498ee265dd832be80: No bootstrap required, opened a new log
I20260810 03:30:54.703785 23064 ts_tablet_manager.cc:1397] T d15dd84e6aa34ce89588cede6d943e8d P 0c4fd1646afb442498ee265dd832be80: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.000s
I20260810 03:30:54.705579 23065 tablet_bootstrap.cc:492] T d15dd84e6aa34ce89588cede6d943e8d P 6a2f5dbb3a4a46ef9050b93425e48df9: No bootstrap required, opened a new log
I20260810 03:30:54.705864 23065 ts_tablet_manager.cc:1397] T d15dd84e6aa34ce89588cede6d943e8d P 6a2f5dbb3a4a46ef9050b93425e48df9: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260810 03:30:54.708530 23070 tablet_bootstrap.cc:492] T d15dd84e6aa34ce89588cede6d943e8d P 8877b25312134dcc8ad309ef4c17f84e: Bootstrap starting.
I20260810 03:30:54.712129 23070 tablet_bootstrap.cc:654] T d15dd84e6aa34ce89588cede6d943e8d P 8877b25312134dcc8ad309ef4c17f84e: Neither blocks nor log segments found. Creating new log.
I20260810 03:30:54.713384 23070 log.cc:824] T d15dd84e6aa34ce89588cede6d943e8d P 8877b25312134dcc8ad309ef4c17f84e: Log is configured to *not* fsync() on all Append() calls
I20260810 03:30:54.713807 23064 raft_consensus.cc:348] T d15dd84e6aa34ce89588cede6d943e8d P 0c4fd1646afb442498ee265dd832be80 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c4fd1646afb442498ee265dd832be80" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 36013 } } peers { permanent_uuid: "6a2f5dbb3a4a46ef9050b93425e48df9" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 40219 } } peers { permanent_uuid: "8877b25312134dcc8ad309ef4c17f84e" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 33847 } }
I20260810 03:30:54.714309 23064 raft_consensus.cc:374] T d15dd84e6aa34ce89588cede6d943e8d P 0c4fd1646afb442498ee265dd832be80 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:30:54.714051 23065 raft_consensus.cc:348] T d15dd84e6aa34ce89588cede6d943e8d P 6a2f5dbb3a4a46ef9050b93425e48df9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c4fd1646afb442498ee265dd832be80" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 36013 } } peers { permanent_uuid: "6a2f5dbb3a4a46ef9050b93425e48df9" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 40219 } } peers { permanent_uuid: "8877b25312134dcc8ad309ef4c17f84e" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 33847 } }
I20260810 03:30:54.714407 23064 raft_consensus.cc:729] T d15dd84e6aa34ce89588cede6d943e8d P 0c4fd1646afb442498ee265dd832be80 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c4fd1646afb442498ee265dd832be80, State: Initialized, Role: FOLLOWER
I20260810 03:30:54.714478 23065 raft_consensus.cc:374] T d15dd84e6aa34ce89588cede6d943e8d P 6a2f5dbb3a4a46ef9050b93425e48df9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:30:54.714572 23065 raft_consensus.cc:729] T d15dd84e6aa34ce89588cede6d943e8d P 6a2f5dbb3a4a46ef9050b93425e48df9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a2f5dbb3a4a46ef9050b93425e48df9, State: Initialized, Role: FOLLOWER
I20260810 03:30:54.715080 23064 consensus_queue.cc:260] T d15dd84e6aa34ce89588cede6d943e8d P 0c4fd1646afb442498ee265dd832be80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c4fd1646afb442498ee265dd832be80" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 36013 } } peers { permanent_uuid: "6a2f5dbb3a4a46ef9050b93425e48df9" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 40219 } } peers { permanent_uuid: "8877b25312134dcc8ad309ef4c17f84e" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 33847 } }
I20260810 03:30:54.715178 23065 consensus_queue.cc:260] T d15dd84e6aa34ce89588cede6d943e8d P 6a2f5dbb3a4a46ef9050b93425e48df9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c4fd1646afb442498ee265dd832be80" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 36013 } } peers { permanent_uuid: "6a2f5dbb3a4a46ef9050b93425e48df9" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 40219 } } peers { permanent_uuid: "8877b25312134dcc8ad309ef4c17f84e" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 33847 } }
I20260810 03:30:54.716459 22824 heartbeater.cc:503] Master 127.27.161.62:45277 was elected leader, sending a full tablet report...
I20260810 03:30:54.716774 23070 tablet_bootstrap.cc:492] T d15dd84e6aa34ce89588cede6d943e8d P 8877b25312134dcc8ad309ef4c17f84e: No bootstrap required, opened a new log
I20260810 03:30:54.717037 23070 ts_tablet_manager.cc:1397] T d15dd84e6aa34ce89588cede6d943e8d P 8877b25312134dcc8ad309ef4c17f84e: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.007s
I20260810 03:30:54.717077 23065 ts_tablet_manager.cc:1428] T d15dd84e6aa34ce89588cede6d943e8d P 6a2f5dbb3a4a46ef9050b93425e48df9: Time spent starting tablet: real 0.011s user 0.008s sys 0.004s
I20260810 03:30:54.717837 23034 heartbeater.cc:503] Master 127.27.161.62:45277 was elected leader, sending a full tablet report...
I20260810 03:30:54.718483 23064 ts_tablet_manager.cc:1428] T d15dd84e6aa34ce89588cede6d943e8d P 0c4fd1646afb442498ee265dd832be80: Time spent starting tablet: real 0.015s user 0.007s sys 0.008s
I20260810 03:30:54.725641 23070 raft_consensus.cc:348] T d15dd84e6aa34ce89588cede6d943e8d P 8877b25312134dcc8ad309ef4c17f84e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c4fd1646afb442498ee265dd832be80" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 36013 } } peers { permanent_uuid: "6a2f5dbb3a4a46ef9050b93425e48df9" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 40219 } } peers { permanent_uuid: "8877b25312134dcc8ad309ef4c17f84e" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 33847 } }
I20260810 03:30:54.726035 23070 raft_consensus.cc:374] T d15dd84e6aa34ce89588cede6d943e8d P 8877b25312134dcc8ad309ef4c17f84e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:30:54.726126 23070 raft_consensus.cc:729] T d15dd84e6aa34ce89588cede6d943e8d P 8877b25312134dcc8ad309ef4c17f84e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8877b25312134dcc8ad309ef4c17f84e, State: Initialized, Role: FOLLOWER
I20260810 03:30:54.726634 23070 consensus_queue.cc:260] T d15dd84e6aa34ce89588cede6d943e8d P 8877b25312134dcc8ad309ef4c17f84e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c4fd1646afb442498ee265dd832be80" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 36013 } } peers { permanent_uuid: "6a2f5dbb3a4a46ef9050b93425e48df9" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 40219 } } peers { permanent_uuid: "8877b25312134dcc8ad309ef4c17f84e" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 33847 } }
I20260810 03:30:54.727977 23070 ts_tablet_manager.cc:1428] T d15dd84e6aa34ce89588cede6d943e8d P 8877b25312134dcc8ad309ef4c17f84e: Time spent starting tablet: real 0.011s user 0.011s sys 0.000s
W20260810 03:30:54.789903 22826 tablet.cc:2369] T d15dd84e6aa34ce89588cede6d943e8d P 6a2f5dbb3a4a46ef9050b93425e48df9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 03:30:54.802556 23073 raft_consensus.cc:482] T d15dd84e6aa34ce89588cede6d943e8d P 6a2f5dbb3a4a46ef9050b93425e48df9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 03:30:54.802887 23073 raft_consensus.cc:504] T d15dd84e6aa34ce89588cede6d943e8d P 6a2f5dbb3a4a46ef9050b93425e48df9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c4fd1646afb442498ee265dd832be80" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 36013 } } peers { permanent_uuid: "6a2f5dbb3a4a46ef9050b93425e48df9" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 40219 } } peers { permanent_uuid: "8877b25312134dcc8ad309ef4c17f84e" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 33847 } }
I20260810 03:30:54.805009 23073 leader_election.cc:288] T d15dd84e6aa34ce89588cede6d943e8d P 6a2f5dbb3a4a46ef9050b93425e48df9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0c4fd1646afb442498ee265dd832be80 (127.27.161.3:36013), 8877b25312134dcc8ad309ef4c17f84e (127.27.161.1:33847)
I20260810 03:30:54.812738 23063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:30:54.806160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.161.2:42149 (local address 127.27.161.1:33847)
0810 03:30:54.806363 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0810 03:30:54.806368 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:30:54.806585 (+ 217us) server_negotiation.cc:407] Connection header received
0810 03:30:54.806823 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:30:54.806827 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:30:54.806885 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:30:54.806957 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:30:54.807647 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:30:54.808728 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:30:54.811034 (+ 2306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:30:54.811808 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:30:54.811967 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:30:54.812047 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0810 03:30:54.812391 (+ 344us) server_negotiation.cc:299] Negotiation successful
0810 03:30:54.812505 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260810 03:30:54.813314 23056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:30:54.805597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.161.2:42091 (local address 127.27.161.3:36013)
0810 03:30:54.805944 (+ 347us) server_negotiation.cc:206] Beginning negotiation
0810 03:30:54.805974 (+ 30us) server_negotiation.cc:399] Waiting for connection header
0810 03:30:54.806306 (+ 332us) server_negotiation.cc:407] Connection header received
0810 03:30:54.806591 (+ 285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:30:54.806598 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:30:54.806667 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:30:54.806723 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:30:54.807574 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:30:54.808337 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:30:54.810897 (+ 2560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:30:54.812081 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:30:54.812260 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:30:54.812333 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0810 03:30:54.812933 (+ 600us) server_negotiation.cc:299] Negotiation successful
0810 03:30:54.813070 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":181}
I20260810 03:30:54.813593 23083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:30:54.805346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.1:33847 (local address 127.27.161.2:42149)
0810 03:30:54.806480 (+ 1134us) negotiation.cc:107] Waiting for socket to connect
0810 03:30:54.806497 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 03:30:54.806682 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:30:54.806999 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:30:54.807004 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:30:54.807044 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:30:54.807438 (+ 394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:30:54.807447 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:30:54.808864 (+ 1417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:30:54.808869 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:30:54.810893 (+ 2024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:30:54.810905 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:30:54.812017 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:30:54.812023 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:30:54.812184 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:30:54.812221 (+ 37us) client_negotiation.cc:769] Sending connection context
0810 03:30:54.813122 (+ 901us) client_negotiation.cc:240] Negotiation successful
0810 03:30:54.813237 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1010,"thread_start_us":1025,"threads_started":1}
I20260810 03:30:54.814064 23081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:30:54.805259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.3:36013 (local address 127.27.161.2:42091)
0810 03:30:54.806180 (+ 921us) negotiation.cc:107] Waiting for socket to connect
0810 03:30:54.806208 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0810 03:30:54.806427 (+ 219us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:30:54.806911 (+ 484us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:30:54.806915 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:30:54.806942 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:30:54.807410 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:30:54.807426 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:30:54.808475 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:30:54.808479 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:30:54.810729 (+ 2250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:30:54.810747 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:30:54.812492 (+ 1745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:30:54.812498 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:30:54.812668 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:30:54.812704 (+ 36us) client_negotiation.cc:769] Sending connection context
0810 03:30:54.813570 (+ 866us) client_negotiation.cc:240] Negotiation successful
0810 03:30:54.813703 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":777,"thread_start_us":534,"threads_started":1}
I20260810 03:30:54.814703 22551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d15dd84e6aa34ce89588cede6d943e8d" candidate_uuid: "6a2f5dbb3a4a46ef9050b93425e48df9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8877b25312134dcc8ad309ef4c17f84e" is_pre_election: true
I20260810 03:30:54.815115 22958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d15dd84e6aa34ce89588cede6d943e8d" candidate_uuid: "6a2f5dbb3a4a46ef9050b93425e48df9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c4fd1646afb442498ee265dd832be80" is_pre_election: true
I20260810 03:30:54.815408 22551 raft_consensus.cc:2454] T d15dd84e6aa34ce89588cede6d943e8d P 8877b25312134dcc8ad309ef4c17f84e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a2f5dbb3a4a46ef9050b93425e48df9 in term 0.
I20260810 03:30:54.815683 22958 raft_consensus.cc:2454] T d15dd84e6aa34ce89588cede6d943e8d P 0c4fd1646afb442498ee265dd832be80 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a2f5dbb3a4a46ef9050b93425e48df9 in term 0.
I20260810 03:30:54.816318 22658 leader_election.cc:302] T d15dd84e6aa34ce89588cede6d943e8d P 6a2f5dbb3a4a46ef9050b93425e48df9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a2f5dbb3a4a46ef9050b93425e48df9, 8877b25312134dcc8ad309ef4c17f84e; no voters:
I20260810 03:30:54.816771 23073 raft_consensus.cc:2781] T d15dd84e6aa34ce89588cede6d943e8d P 6a2f5dbb3a4a46ef9050b93425e48df9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 03:30:54.816932 23073 raft_consensus.cc:482] T d15dd84e6aa34ce89588cede6d943e8d P 6a2f5dbb3a4a46ef9050b93425e48df9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 03:30:54.817024 23073 raft_consensus.cc:3037] T d15dd84e6aa34ce89588cede6d943e8d P 6a2f5dbb3a4a46ef9050b93425e48df9 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:30:54.819515 23073 raft_consensus.cc:504] T d15dd84e6aa34ce89588cede6d943e8d P 6a2f5dbb3a4a46ef9050b93425e48df9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c4fd1646afb442498ee265dd832be80" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 36013 } } peers { permanent_uuid: "6a2f5dbb3a4a46ef9050b93425e48df9" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 40219 } } peers { permanent_uuid: "8877b25312134dcc8ad309ef4c17f84e" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 33847 } }
I20260810 03:30:54.820108 23073 leader_election.cc:288] T d15dd84e6aa34ce89588cede6d943e8d P 6a2f5dbb3a4a46ef9050b93425e48df9 [CANDIDATE]: Term 1 election: Requested vote from peers 0c4fd1646afb442498ee265dd832be80 (127.27.161.3:36013), 8877b25312134dcc8ad309ef4c17f84e (127.27.161.1:33847)
I20260810 03:30:54.820529 22958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d15dd84e6aa34ce89588cede6d943e8d" candidate_uuid: "6a2f5dbb3a4a46ef9050b93425e48df9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c4fd1646afb442498ee265dd832be80"
I20260810 03:30:54.820533 22551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d15dd84e6aa34ce89588cede6d943e8d" candidate_uuid: "6a2f5dbb3a4a46ef9050b93425e48df9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8877b25312134dcc8ad309ef4c17f84e"
I20260810 03:30:54.820699 22958 raft_consensus.cc:3037] T d15dd84e6aa34ce89588cede6d943e8d P 0c4fd1646afb442498ee265dd832be80 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:30:54.820729 22551 raft_consensus.cc:3037] T d15dd84e6aa34ce89588cede6d943e8d P 8877b25312134dcc8ad309ef4c17f84e [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:30:54.822924 22958 raft_consensus.cc:2454] T d15dd84e6aa34ce89588cede6d943e8d P 0c4fd1646afb442498ee265dd832be80 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a2f5dbb3a4a46ef9050b93425e48df9 in term 1.
I20260810 03:30:54.822927 22551 raft_consensus.cc:2454] T d15dd84e6aa34ce89588cede6d943e8d P 8877b25312134dcc8ad309ef4c17f84e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a2f5dbb3a4a46ef9050b93425e48df9 in term 1.
I20260810 03:30:54.823354 22660 leader_election.cc:302] T d15dd84e6aa34ce89588cede6d943e8d P 6a2f5dbb3a4a46ef9050b93425e48df9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0c4fd1646afb442498ee265dd832be80, 6a2f5dbb3a4a46ef9050b93425e48df9, 8877b25312134dcc8ad309ef4c17f84e; no voters:
I20260810 03:30:54.823679 23073 raft_consensus.cc:2781] T d15dd84e6aa34ce89588cede6d943e8d P 6a2f5dbb3a4a46ef9050b93425e48df9 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:30:54.824110 23073 raft_consensus.cc:686] T d15dd84e6aa34ce89588cede6d943e8d P 6a2f5dbb3a4a46ef9050b93425e48df9 [term 1 LEADER]: Becoming Leader. State: Replica: 6a2f5dbb3a4a46ef9050b93425e48df9, State: Running, Role: LEADER
I20260810 03:30:54.824568 23073 consensus_queue.cc:237] T d15dd84e6aa34ce89588cede6d943e8d P 6a2f5dbb3a4a46ef9050b93425e48df9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c4fd1646afb442498ee265dd832be80" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 36013 } } peers { permanent_uuid: "6a2f5dbb3a4a46ef9050b93425e48df9" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 40219 } } peers { permanent_uuid: "8877b25312134dcc8ad309ef4c17f84e" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 33847 } }
I20260810 03:30:54.828867 22314 catalog_manager.cc:5495] T d15dd84e6aa34ce89588cede6d943e8d P 6a2f5dbb3a4a46ef9050b93425e48df9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6a2f5dbb3a4a46ef9050b93425e48df9 (127.27.161.2). New cstate: current_term: 1 leader_uuid: "6a2f5dbb3a4a46ef9050b93425e48df9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c4fd1646afb442498ee265dd832be80" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 36013 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6a2f5dbb3a4a46ef9050b93425e48df9" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 40219 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8877b25312134dcc8ad309ef4c17f84e" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 33847 } health_report { overall_health: UNKNOWN } } }
W20260810 03:30:54.837532 23036 tablet.cc:2369] T d15dd84e6aa34ce89588cede6d943e8d P 0c4fd1646afb442498ee265dd832be80: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 03:30:54.863672 23058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:30:54.858139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58616 (local address 127.27.161.2:40219)
0810 03:30:54.858365 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0810 03:30:54.858371 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:30:54.858391 (+ 20us) server_negotiation.cc:407] Connection header received
0810 03:30:54.858489 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:30:54.858493 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:30:54.858584 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:30:54.858723 (+ 139us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 03:30:54.859028 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:30:54.859777 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:30:54.861233 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:30:54.861696 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:30:54.861804 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:30:54.862448 (+ 644us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 03:30:54.863053 (+ 605us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 03:30:54.863167 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0810 03:30:54.863292 (+ 125us) server_negotiation.cc:299] Negotiation successful
0810 03:30:54.863371 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260810 03:30:54.872453 22551 raft_consensus.cc:1261] T d15dd84e6aa34ce89588cede6d943e8d P 8877b25312134dcc8ad309ef4c17f84e [term 1 FOLLOWER]: Refusing update from remote peer 6a2f5dbb3a4a46ef9050b93425e48df9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:30:54.872467 22958 raft_consensus.cc:1261] T d15dd84e6aa34ce89588cede6d943e8d P 0c4fd1646afb442498ee265dd832be80 [term 1 FOLLOWER]: Refusing update from remote peer 6a2f5dbb3a4a46ef9050b93425e48df9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:30:54.873512 23084 consensus_queue.cc:1035] T d15dd84e6aa34ce89588cede6d943e8d P 6a2f5dbb3a4a46ef9050b93425e48df9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8877b25312134dcc8ad309ef4c17f84e" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 33847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:30:54.873942 23073 consensus_queue.cc:1035] T d15dd84e6aa34ce89588cede6d943e8d P 6a2f5dbb3a4a46ef9050b93425e48df9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c4fd1646afb442498ee265dd832be80" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 36013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:30:54.905530 22315 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45010:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 03:30:54.905850 22315 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45010:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 03:30:54.909582 22315 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2aa7e5a0a00e431bb6a609dcab5c49d4: Sending DeleteTablet for 3 replicas of tablet d15dd84e6aa34ce89588cede6d943e8d
I20260810 03:30:54.910871 22931 tablet_service.cc:1514] Processing DeleteTablet for tablet d15dd84e6aa34ce89588cede6d943e8d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:30:54 UTC) from {username='kudu', principal='kudu/127.27.161.62@KRBTEST.COM'} at 127.0.0.1:48404
I20260810 03:30:54.911001 22509 tablet_service.cc:1514] Processing DeleteTablet for tablet d15dd84e6aa34ce89588cede6d943e8d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:30:54 UTC) from {username='kudu', principal='kudu/127.27.161.62@KRBTEST.COM'} at 127.0.0.1:51410
I20260810 03:30:54.911144 22723 tablet_service.cc:1514] Processing DeleteTablet for tablet d15dd84e6aa34ce89588cede6d943e8d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:30:54 UTC) from {username='kudu', principal='kudu/127.27.161.62@KRBTEST.COM'} at 127.0.0.1:58606
I20260810 03:30:54.913648 28292 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iBH5/build/asan/bin/kudu pid 22406
I20260810 03:30:54.924142 23099 tablet_replica.cc:330] T d15dd84e6aa34ce89588cede6d943e8d P 8877b25312134dcc8ad309ef4c17f84e: stopping tablet replica
I20260810 03:30:54.924743 23099 raft_consensus.cc:2229] T d15dd84e6aa34ce89588cede6d943e8d P 8877b25312134dcc8ad309ef4c17f84e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 03:30:54.925423 23099 raft_consensus.cc:2258] T d15dd84e6aa34ce89588cede6d943e8d P 8877b25312134dcc8ad309ef4c17f84e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:30:54.928224 23099 ts_tablet_manager.cc:1905] T d15dd84e6aa34ce89588cede6d943e8d P 8877b25312134dcc8ad309ef4c17f84e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:30:54.936589 23099 ts_tablet_manager.cc:1918] T d15dd84e6aa34ce89588cede6d943e8d P 8877b25312134dcc8ad309ef4c17f84e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 03:30:54.936846 23099 log.cc:1196] T d15dd84e6aa34ce89588cede6d943e8d P 8877b25312134dcc8ad309ef4c17f84e: Deleting WAL directory at /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786332336319544-28292-0/minicluster-data/ts-0/wal/wals/d15dd84e6aa34ce89588cede6d943e8d
I20260810 03:30:54.937489 23099 ts_tablet_manager.cc:1939] T d15dd84e6aa34ce89588cede6d943e8d P 8877b25312134dcc8ad309ef4c17f84e: Deleting consensus metadata
I20260810 03:30:54.938760 23100 tablet_replica.cc:330] T d15dd84e6aa34ce89588cede6d943e8d P 0c4fd1646afb442498ee265dd832be80: stopping tablet replica
I20260810 03:30:54.939296 23100 raft_consensus.cc:2229] T d15dd84e6aa34ce89588cede6d943e8d P 0c4fd1646afb442498ee265dd832be80 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 03:30:54.939893 23100 raft_consensus.cc:2258] T d15dd84e6aa34ce89588cede6d943e8d P 0c4fd1646afb442498ee265dd832be80 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:30:54.940901 22292 catalog_manager.cc:4841] TS 8877b25312134dcc8ad309ef4c17f84e (127.27.161.1:33847): tablet d15dd84e6aa34ce89588cede6d943e8d (table test-table [id=9dce2492570c4998ae56b26786508848]) successfully deleted
I20260810 03:30:54.942390 23100 ts_tablet_manager.cc:1905] T d15dd84e6aa34ce89588cede6d943e8d P 0c4fd1646afb442498ee265dd832be80: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:30:54.942498 23101 tablet_replica.cc:330] T d15dd84e6aa34ce89588cede6d943e8d P 6a2f5dbb3a4a46ef9050b93425e48df9: stopping tablet replica
I20260810 03:30:54.942963 23101 raft_consensus.cc:2229] T d15dd84e6aa34ce89588cede6d943e8d P 6a2f5dbb3a4a46ef9050b93425e48df9 [term 1 LEADER]: Raft consensus shutting down.
I20260810 03:30:54.943677 23101 raft_consensus.cc:2258] T d15dd84e6aa34ce89588cede6d943e8d P 6a2f5dbb3a4a46ef9050b93425e48df9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:30:54.946097 23101 ts_tablet_manager.cc:1905] T d15dd84e6aa34ce89588cede6d943e8d P 6a2f5dbb3a4a46ef9050b93425e48df9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:30:54.949916 23100 ts_tablet_manager.cc:1918] T d15dd84e6aa34ce89588cede6d943e8d P 0c4fd1646afb442498ee265dd832be80: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 03:30:54.950199 23100 log.cc:1196] T d15dd84e6aa34ce89588cede6d943e8d P 0c4fd1646afb442498ee265dd832be80: Deleting WAL directory at /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786332336319544-28292-0/minicluster-data/ts-2/wal/wals/d15dd84e6aa34ce89588cede6d943e8d
I20260810 03:30:54.950790 23100 ts_tablet_manager.cc:1939] T d15dd84e6aa34ce89588cede6d943e8d P 0c4fd1646afb442498ee265dd832be80: Deleting consensus metadata
Aug 10 03:30:54 dist-test-slave-vpvm krb5kdc[22243](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332652, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.161.1@KRBTEST.COM
I20260810 03:30:54.952788 22296 catalog_manager.cc:4841] TS 0c4fd1646afb442498ee265dd832be80 (127.27.161.3:36013): tablet d15dd84e6aa34ce89588cede6d943e8d (table test-table [id=9dce2492570c4998ae56b26786508848]) successfully deleted
I20260810 03:30:54.953456 23101 ts_tablet_manager.cc:1918] T d15dd84e6aa34ce89588cede6d943e8d P 6a2f5dbb3a4a46ef9050b93425e48df9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 03:30:54.953613 23101 log.cc:1196] T d15dd84e6aa34ce89588cede6d943e8d P 6a2f5dbb3a4a46ef9050b93425e48df9: Deleting WAL directory at /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786332336319544-28292-0/minicluster-data/ts-1/wal/wals/d15dd84e6aa34ce89588cede6d943e8d
I20260810 03:30:54.954195 23101 ts_tablet_manager.cc:1939] T d15dd84e6aa34ce89588cede6d943e8d P 6a2f5dbb3a4a46ef9050b93425e48df9: Deleting consensus metadata
I20260810 03:30:54.955847 22290 catalog_manager.cc:4841] TS 6a2f5dbb3a4a46ef9050b93425e48df9 (127.27.161.2:40219): tablet d15dd84e6aa34ce89588cede6d943e8d (table test-table [id=9dce2492570c4998ae56b26786508848]) successfully deleted
I20260810 03:30:54.959816 23063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:30:54.914552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51422 (local address 127.27.161.1:33847)
0810 03:30:54.914744 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0810 03:30:54.914751 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:30:54.939538 (+ 24787us) server_negotiation.cc:407] Connection header received
0810 03:30:54.940197 (+ 659us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:30:54.940203 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:30:54.940308 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:30:54.940386 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:30:54.942374 (+ 1988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:30:54.943523 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:30:54.945914 (+ 2391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:30:54.946602 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:30:54.946723 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:30:54.952756 (+ 6033us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:30:54.952778 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:30:54.952800 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:30:54.952849 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:30:54.956820 (+ 3971us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:30:54.957563 (+ 743us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:30:54.957573 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:30:54.957577 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:30:54.957787 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:30:54.958179 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:30:54.958185 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:30:54.958189 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:30:54.958659 (+ 470us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:30:54.958999 (+ 340us) server_negotiation.cc:1035] Waiting for connection context
0810 03:30:54.959168 (+ 169us) server_negotiation.cc:299] Negotiation successful
0810 03:30:54.959405 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260810 03:30:54.960476 22613 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:30:55.059528 28292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iBH5/build/asan/bin/kudu with pid 22406
I20260810 03:30:55.082518 28292 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iBH5/build/asan/bin/kudu pid 22631
Aug 10 03:30:55 dist-test-slave-vpvm krb5kdc[22243](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332652, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.161.2@KRBTEST.COM
I20260810 03:30:55.098920 23058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:30:55.083401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58630 (local address 127.27.161.2:40219)
0810 03:30:55.083615 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0810 03:30:55.083621 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:30:55.083641 (+ 20us) server_negotiation.cc:407] Connection header received
0810 03:30:55.083719 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:30:55.083724 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:30:55.083798 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:30:55.083882 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:30:55.085154 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:30:55.086098 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:30:55.087454 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:30:55.087908 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:30:55.088032 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:30:55.092788 (+ 4756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:30:55.092807 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:30:55.092826 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:30:55.092870 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:30:55.095946 (+ 3076us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:30:55.096698 (+ 752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:30:55.096714 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:30:55.096716 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:30:55.096881 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:30:55.097260 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:30:55.097264 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:30:55.097266 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:30:55.097634 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:30:55.098037 (+ 403us) server_negotiation.cc:1035] Waiting for connection context
0810 03:30:55.098285 (+ 248us) server_negotiation.cc:299] Negotiation successful
0810 03:30:55.098535 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260810 03:30:55.099737 22818 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:30:55.215540 28292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iBH5/build/asan/bin/kudu with pid 22631
I20260810 03:30:55.241072 28292 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iBH5/build/asan/bin/kudu pid 22833
Aug 10 03:30:55 dist-test-slave-vpvm krb5kdc[22243](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332652, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.161.3@KRBTEST.COM
I20260810 03:30:55.256471 23056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:30:55.241809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48408 (local address 127.27.161.3:36013)
0810 03:30:55.241995 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0810 03:30:55.242000 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:30:55.242042 (+ 42us) server_negotiation.cc:407] Connection header received
0810 03:30:55.242209 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:30:55.242213 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:30:55.242282 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:30:55.242368 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:30:55.243728 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:30:55.244546 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:30:55.246250 (+ 1704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:30:55.246671 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:30:55.246770 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:30:55.251218 (+ 4448us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:30:55.251236 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:30:55.251255 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:30:55.251299 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:30:55.254023 (+ 2724us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:30:55.254695 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:30:55.254702 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:30:55.254705 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:30:55.254851 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:30:55.255156 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:30:55.255160 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:30:55.255162 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:30:55.255564 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:30:55.255788 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0810 03:30:55.256022 (+ 234us) server_negotiation.cc:299] Negotiation successful
0810 03:30:55.256199 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260810 03:30:55.257098 23028 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:30:55.346634 28292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iBH5/build/asan/bin/kudu with pid 22833
I20260810 03:30:55.374083 28292 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iBH5/build/asan/bin/kudu pid 22264
I20260810 03:30:55.375554 22356 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:30:55.508119 28292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iBH5/build/asan/bin/kudu with pid 22264
2026-08-10T03:30:55Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (4719 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-08-10T03:30:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T03:30:55Z Disabled control of system clock
I20260810 03:30:55.616672 28292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iBH5/build/asan/bin/kudu
/tmp/dist-test-task24iBH5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786332336319544-28292-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786332336319544-28292-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786332336319544-28292-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786332336319544-28292-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.161.62:33913
--webserver_interface=127.27.161.62
--webserver_port=0
--builtin_ntp_servers=127.27.161.20:38321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.161.62:33913
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260810 03:30:55.982323 23116 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:30:55.982824 23116 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:30:55.982946 23116 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:30:55.991050 23116 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:30:55.991187 23116 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:30:55.991235 23116 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:30:55.991303 23116 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:30:55.991348 23116 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:30:56.001557 23116 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.161.20:38321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786332336319544-28292-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786332336319544-28292-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.161.62:33913
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.161.62:33913
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786332336319544-28292-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.161.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786332336319544-28292-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:30:56.003011 23116 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:30:56.005131 23116 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:30:56.017904 23130 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:30:56.017928 23135 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:30:56.017876 23132 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:30:56.018844 23116 server_base.cc:1034] running on GCE node
I20260810 03:30:56.019727 23116 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:30:56.021406 23116 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:30:56.022660 23116 hybrid_clock.cc:648] HybridClock initialized: now 1786332656022595 us; error 58 us; skew 500 ppm
I20260810 03:30:56.023092 23116 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 03:30:56.029507 23116 webserver.cc:460] Webserver started at http://127.27.161.62:35589/ using document root <none> and password file <none>
I20260810 03:30:56.030458 23116 fs_manager.cc:359] Metadata directory not provided
I20260810 03:30:56.030601 23116 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:30:56.030973 23116 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:30:56.034206 23116 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786332336319544-28292-0/minicluster-data/master-0/data/instance:
uuid: "37d955abfde248809ce4ee9e5e39fe79"
format_stamp: "Formatted at 2026-08-10 03:30:56 on dist-test-slave-vpvm"
server_key: "2b733463af6ab846132e6f54dc909b77"
server_key_iv: "c8ad8396a7625fa7d7384cad95e0e399"
server_key_version: "encryptionkey@0"
I20260810 03:30:56.035087 23116 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786332336319544-28292-0/minicluster-data/master-0/wal/instance:
uuid: "37d955abfde248809ce4ee9e5e39fe79"
format_stamp: "Formatted at 2026-08-10 03:30:56 on dist-test-slave-vpvm"
server_key: "2b733463af6ab846132e6f54dc909b77"
server_key_iv: "c8ad8396a7625fa7d7384cad95e0e399"
server_key_version: "encryptionkey@0"
I20260810 03:30:56.040851 23116 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260810 03:30:56.044904 23142 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:30:56.046756 23116 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260810 03:30:56.046981 23116 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786332336319544-28292-0/minicluster-data/master-0/data,/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786332336319544-28292-0/minicluster-data/master-0/wal
uuid: "37d955abfde248809ce4ee9e5e39fe79"
format_stamp: "Formatted at 2026-08-10 03:30:56 on dist-test-slave-vpvm"
server_key: "2b733463af6ab846132e6f54dc909b77"
server_key_iv: "c8ad8396a7625fa7d7384cad95e0e399"
server_key_version: "encryptionkey@0"
I20260810 03:30:56.047245 23116 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786332336319544-28292-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786332336319544-28292-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786332336319544-28292-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:30:56.060431 23116 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:30:56.061461 23116 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:30:56.061779 23116 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:30:56.081161 23116 rpc_server.cc:307] RPC server started. Bound to: 127.27.161.62:33913
I20260810 03:30:56.081184 23222 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.161.62:33913 every 8 connection(s)
I20260810 03:30:56.082661 23116 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786332336319544-28292-0/minicluster-data/master-0/data/info.pb
I20260810 03:30:56.086835 23224 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:30:56.091962 28292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iBH5/build/asan/bin/kudu as pid 23116
I20260810 03:30:56.092149 28292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786332336319544-28292-0/minicluster-data/master-0/wal/instance
I20260810 03:30:56.092602 28292 external_mini_cluster.cc:1428] Setting key 01591e498540926c3904457ef6bab15d
I20260810 03:30:56.100028 23224 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 37d955abfde248809ce4ee9e5e39fe79: Bootstrap starting.
I20260810 03:30:56.104621 23224 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 37d955abfde248809ce4ee9e5e39fe79: Neither blocks nor log segments found. Creating new log.
I20260810 03:30:56.106218 23224 log.cc:824] T 00000000000000000000000000000000 P 37d955abfde248809ce4ee9e5e39fe79: Log is configured to *not* fsync() on all Append() calls
I20260810 03:30:56.110124 23228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:30:56.094717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43956 (local address 127.27.161.62:33913)
0810 03:30:56.095887 (+ 1170us) server_negotiation.cc:206] Beginning negotiation
0810 03:30:56.095917 (+ 30us) server_negotiation.cc:399] Waiting for connection header
0810 03:30:56.096012 (+ 95us) server_negotiation.cc:407] Connection header received
0810 03:30:56.098517 (+ 2505us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:30:56.098553 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:30:56.099273 (+ 720us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:30:56.099758 (+ 485us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:30:56.100370 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:30:56.102397 (+ 2027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:30:56.104419 (+ 2022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:30:56.105596 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:30:56.105819 (+ 223us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:30:56.107102 (+ 1283us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:30:56.107111 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:30:56.107128 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:30:56.107766 (+ 638us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:30:56.107876 (+ 110us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:30:56.107907 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:30:56.108041 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0810 03:30:56.108278 (+ 237us) server_negotiation.cc:299] Negotiation successful
0810 03:30:56.108654 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":605,"thread_start_us":372,"threads_started":1}
I20260810 03:30:56.110134 23224 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 37d955abfde248809ce4ee9e5e39fe79: No bootstrap required, opened a new log
I20260810 03:30:56.118664 23224 raft_consensus.cc:348] T 00000000000000000000000000000000 P 37d955abfde248809ce4ee9e5e39fe79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37d955abfde248809ce4ee9e5e39fe79" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 33913 } }
I20260810 03:30:56.119096 23224 raft_consensus.cc:374] T 00000000000000000000000000000000 P 37d955abfde248809ce4ee9e5e39fe79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:30:56.119176 23224 raft_consensus.cc:729] T 00000000000000000000000000000000 P 37d955abfde248809ce4ee9e5e39fe79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37d955abfde248809ce4ee9e5e39fe79, State: Initialized, Role: FOLLOWER
I20260810 03:30:56.119765 23224 consensus_queue.cc:260] T 00000000000000000000000000000000 P 37d955abfde248809ce4ee9e5e39fe79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37d955abfde248809ce4ee9e5e39fe79" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 33913 } }
I20260810 03:30:56.120072 23224 raft_consensus.cc:388] T 00000000000000000000000000000000 P 37d955abfde248809ce4ee9e5e39fe79 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 03:30:56.120220 23224 raft_consensus.cc:482] T 00000000000000000000000000000000 P 37d955abfde248809ce4ee9e5e39fe79 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 03:30:56.120419 23224 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 37d955abfde248809ce4ee9e5e39fe79 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:30:56.122994 23224 raft_consensus.cc:504] T 00000000000000000000000000000000 P 37d955abfde248809ce4ee9e5e39fe79 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37d955abfde248809ce4ee9e5e39fe79" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 33913 } }
I20260810 03:30:56.123677 23224 leader_election.cc:302] T 00000000000000000000000000000000 P 37d955abfde248809ce4ee9e5e39fe79 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 37d955abfde248809ce4ee9e5e39fe79; no voters:
I20260810 03:30:56.124303 23224 leader_election.cc:288] T 00000000000000000000000000000000 P 37d955abfde248809ce4ee9e5e39fe79 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 03:30:56.124460 23230 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 37d955abfde248809ce4ee9e5e39fe79 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:30:56.125212 23230 raft_consensus.cc:686] T 00000000000000000000000000000000 P 37d955abfde248809ce4ee9e5e39fe79 [term 1 LEADER]: Becoming Leader. State: Replica: 37d955abfde248809ce4ee9e5e39fe79, State: Running, Role: LEADER
I20260810 03:30:56.125798 23230 consensus_queue.cc:237] T 00000000000000000000000000000000 P 37d955abfde248809ce4ee9e5e39fe79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37d955abfde248809ce4ee9e5e39fe79" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 33913 } }
I20260810 03:30:56.126487 23224 sys_catalog.cc:564] T 00000000000000000000000000000000 P 37d955abfde248809ce4ee9e5e39fe79 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:30:56.128846 23231 sys_catalog.cc:455] T 00000000000000000000000000000000 P 37d955abfde248809ce4ee9e5e39fe79 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "37d955abfde248809ce4ee9e5e39fe79" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37d955abfde248809ce4ee9e5e39fe79" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 33913 } } }
I20260810 03:30:56.129328 23231 sys_catalog.cc:458] T 00000000000000000000000000000000 P 37d955abfde248809ce4ee9e5e39fe79 [sys.catalog]: This master's current role is: LEADER
I20260810 03:30:56.130162 23233 sys_catalog.cc:455] T 00000000000000000000000000000000 P 37d955abfde248809ce4ee9e5e39fe79 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 37d955abfde248809ce4ee9e5e39fe79. Latest consensus state: current_term: 1 leader_uuid: "37d955abfde248809ce4ee9e5e39fe79" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37d955abfde248809ce4ee9e5e39fe79" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 33913 } } }
I20260810 03:30:56.130470 23233 sys_catalog.cc:458] T 00000000000000000000000000000000 P 37d955abfde248809ce4ee9e5e39fe79 [sys.catalog]: This master's current role is: LEADER
I20260810 03:30:56.134826 23240 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 03:30:56.141435 23240 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 03:30:56.156855 23240 catalog_manager.cc:1347] Generated new cluster ID: 9f1df6a607d84ee3ab22ad5c5b339178
I20260810 03:30:56.156987 23240 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 03:30:56.169574 23240 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 03:30:56.171010 23240 catalog_manager.cc:1504] Loading token signing keys...
I20260810 03:30:56.178406 23240 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 37d955abfde248809ce4ee9e5e39fe79: Generated new TSK 0
I20260810 03:30:56.179415 23240 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 03:30:56.184757 28292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iBH5/build/asan/bin/kudu
/tmp/dist-test-task24iBH5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786332336319544-28292-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786332336319544-28292-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786332336319544-28292-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786332336319544-28292-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.161.1:0
--local_ip_for_outbound_sockets=127.27.161.1
--webserver_interface=127.27.161.1
--webserver_port=0
--tserver_master_addrs=127.27.161.62:33913
--builtin_ntp_servers=127.27.161.20:38321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260810 03:30:56.558998 23260 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:30:56.559473 23260 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:30:56.559650 23260 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:30:56.567920 23260 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:30:56.568049 23260 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:30:56.568171 23260 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.161.1
I20260810 03:30:56.578181 23260 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.161.20:38321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786332336319544-28292-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786332336319544-28292-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.161.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786332336319544-28292-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.161.1
--webserver_port=0
--tserver_master_addrs=127.27.161.62:33913
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.161.1
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786332336319544-28292-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:30:56.579599 23260 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:30:56.581507 23260 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:30:56.595788 23268 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:30:56.595826 23272 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:30:56.595855 23269 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:30:56.596863 23260 server_base.cc:1034] running on GCE node
I20260810 03:30:56.597554 23260 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:30:56.598675 23260 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:30:56.599922 23260 hybrid_clock.cc:648] HybridClock initialized: now 1786332656599851 us; error 60 us; skew 500 ppm
I20260810 03:30:56.600263 23260 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 03:30:56.602788 23260 webserver.cc:460] Webserver started at http://127.27.161.1:37317/ using document root <none> and password file <none>
I20260810 03:30:56.603549 23260 fs_manager.cc:359] Metadata directory not provided
I20260810 03:30:56.603662 23260 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:30:56.603995 23260 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:30:56.607216 23260 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786332336319544-28292-0/minicluster-data/ts-0/data/instance:
uuid: "2414144a3e51404ebbbad2733efe8c29"
format_stamp: "Formatted at 2026-08-10 03:30:56 on dist-test-slave-vpvm"
server_key: "e373a77cdff1b8417e3207a188f10a0c"
server_key_iv: "0eac303a731c6066085d1554fb73d464"
server_key_version: "encryptionkey@0"
I20260810 03:30:56.608103 23260 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786332336319544-28292-0/minicluster-data/ts-0/wal/instance:
uuid: "2414144a3e51404ebbbad2733efe8c29"
format_stamp: "Formatted at 2026-08-10 03:30:56 on dist-test-slave-vpvm"
server_key: "e373a77cdff1b8417e3207a188f10a0c"
server_key_iv: "0eac303a731c6066085d1554fb73d464"
server_key_version: "encryptionkey@0"
I20260810 03:30:56.613899 23260 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260810 03:30:56.617810 23279 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:30:56.619398 23260 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 03:30:56.619589 23260 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786332336319544-28292-0/minicluster-data/ts-0/data,/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786332336319544-28292-0/minicluster-data/ts-0/wal
uuid: "2414144a3e51404ebbbad2733efe8c29"
format_stamp: "Formatted at 2026-08-10 03:30:56 on dist-test-slave-vpvm"
server_key: "e373a77cdff1b8417e3207a188f10a0c"
server_key_iv: "0eac303a731c6066085d1554fb73d464"
server_key_version: "encryptionkey@0"
I20260810 03:30:56.619823 23260 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786332336319544-28292-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786332336319544-28292-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786332336319544-28292-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:30:56.653303 23260 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:30:56.654312 23260 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:30:56.654623 23260 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:30:56.655776 23260 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:30:56.657439 23260 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:30:56.657549 23260 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:30:56.657663 23260 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:30:56.657761 23260 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:30:56.690809 23260 rpc_server.cc:307] RPC server started. Bound to: 127.27.161.1:44423
I20260810 03:30:56.690901 23443 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.161.1:44423 every 8 connection(s)
I20260810 03:30:56.692310 23260 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786332336319544-28292-0/minicluster-data/ts-0/data/info.pb
I20260810 03:30:56.700403 28292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iBH5/build/asan/bin/kudu as pid 23260
I20260810 03:30:56.700590 28292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786332336319544-28292-0/minicluster-data/ts-0/wal/instance
I20260810 03:30:56.701046 28292 external_mini_cluster.cc:1428] Setting key c9598d56f5db926b54182d8ba2db2026
I20260810 03:30:56.704134 28292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iBH5/build/asan/bin/kudu
/tmp/dist-test-task24iBH5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786332336319544-28292-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786332336319544-28292-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786332336319544-28292-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786332336319544-28292-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.161.2:0
--local_ip_for_outbound_sockets=127.27.161.2
--webserver_interface=127.27.161.2
--webserver_port=0
--tserver_master_addrs=127.27.161.62:33913
--builtin_ntp_servers=127.27.161.20:38321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260810 03:30:56.705752 23448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:30:56.695041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.161.1:36879 (local address 127.27.161.62:33913)
0810 03:30:56.695503 (+ 462us) server_negotiation.cc:206] Beginning negotiation
0810 03:30:56.695508 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:30:56.696643 (+ 1135us) server_negotiation.cc:407] Connection header received
0810 03:30:56.698647 (+ 2004us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:30:56.698652 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:30:56.698742 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:30:56.698848 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:30:56.700075 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:30:56.700873 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:30:56.702932 (+ 2059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:30:56.703358 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:30:56.703476 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:30:56.704889 (+ 1413us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:30:56.704894 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:30:56.704898 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:30:56.704916 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:30:56.704958 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:30:56.704971 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:30:56.705046 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0810 03:30:56.705282 (+ 236us) server_negotiation.cc:299] Negotiation successful
0810 03:30:56.705392 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":226,"threads_started":1}
I20260810 03:30:56.706892 23450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:30:56.695526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:33913 (local address 127.27.161.1:36879)
0810 03:30:56.696459 (+ 933us) negotiation.cc:107] Waiting for socket to connect
0810 03:30:56.696507 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0810 03:30:56.698319 (+ 1812us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:30:56.699131 (+ 812us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:30:56.699140 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:30:56.699242 (+ 102us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:30:56.699878 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:30:56.699897 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:30:56.701033 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:30:56.701037 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:30:56.702784 (+ 1747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:30:56.702802 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:30:56.703599 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:30:56.703608 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:30:56.703847 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:30:56.704536 (+ 689us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 03:30:56.704539 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:30:56.704585 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 03:30:56.704688 (+ 103us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:30:56.705063 (+ 375us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:30:56.705075 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:30:56.705084 (+ 9us) client_negotiation.cc:769] Sending connection context
0810 03:30:56.705256 (+ 172us) client_negotiation.cc:240] Negotiation successful
0810 03:30:56.705716 (+ 460us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":482,"thread_start_us":290,"threads_started":1}
I20260810 03:30:56.708585 23446 heartbeater.cc:347] Connected to a master server at 127.27.161.62:33913
I20260810 03:30:56.708894 23446 heartbeater.cc:464] Registering TS with master...
I20260810 03:30:56.709681 23446 heartbeater.cc:511] Master 127.27.161.62:33913 requested a full tablet report, sending...
I20260810 03:30:56.712028 23167 ts_manager.cc:194] Registered new tserver with Master: 2414144a3e51404ebbbad2733efe8c29 (127.27.161.1:44423)
I20260810 03:30:56.713984 23167 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.161.1:36879
W20260810 03:30:57.066748 23452 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:30:57.067093 23452 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:30:57.067229 23452 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:30:57.074653 23452 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:30:57.074781 23452 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:30:57.074932 23452 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.161.2
I20260810 03:30:57.084622 23452 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.161.20:38321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786332336319544-28292-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786332336319544-28292-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.161.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786332336319544-28292-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.161.2
--webserver_port=0
--tserver_master_addrs=127.27.161.62:33913
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.161.2
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786332336319544-28292-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:30:57.085969 23452 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:30:57.087848 23452 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:30:57.101219 23460 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:30:57.101284 23465 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:30:57.101243 23462 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:30:57.102255 23452 server_base.cc:1034] running on GCE node
I20260810 03:30:57.102862 23452 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:30:57.103845 23452 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:30:57.105049 23452 hybrid_clock.cc:648] HybridClock initialized: now 1786332657104986 us; error 50 us; skew 500 ppm
I20260810 03:30:57.105371 23452 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 03:30:57.107796 23452 webserver.cc:460] Webserver started at http://127.27.161.2:33553/ using document root <none> and password file <none>
I20260810 03:30:57.108523 23452 fs_manager.cc:359] Metadata directory not provided
I20260810 03:30:57.108619 23452 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:30:57.108906 23452 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:30:57.112087 23452 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786332336319544-28292-0/minicluster-data/ts-1/data/instance:
uuid: "e7c5365c5e98448eba0ba5683f057cea"
format_stamp: "Formatted at 2026-08-10 03:30:57 on dist-test-slave-vpvm"
server_key: "c4440bcc8f1551941348380913ebeb25"
server_key_iv: "3959d2caed4215bfd7f04fd4438eaaca"
server_key_version: "encryptionkey@0"
I20260810 03:30:57.112883 23452 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786332336319544-28292-0/minicluster-data/ts-1/wal/instance:
uuid: "e7c5365c5e98448eba0ba5683f057cea"
format_stamp: "Formatted at 2026-08-10 03:30:57 on dist-test-slave-vpvm"
server_key: "c4440bcc8f1551941348380913ebeb25"
server_key_iv: "3959d2caed4215bfd7f04fd4438eaaca"
server_key_version: "encryptionkey@0"
I20260810 03:30:57.118652 23452 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260810 03:30:57.122449 23474 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:30:57.123986 23452 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260810 03:30:57.124182 23452 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786332336319544-28292-0/minicluster-data/ts-1/data,/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786332336319544-28292-0/minicluster-data/ts-1/wal
uuid: "e7c5365c5e98448eba0ba5683f057cea"
format_stamp: "Formatted at 2026-08-10 03:30:57 on dist-test-slave-vpvm"
server_key: "c4440bcc8f1551941348380913ebeb25"
server_key_iv: "3959d2caed4215bfd7f04fd4438eaaca"
server_key_version: "encryptionkey@0"
I20260810 03:30:57.124403 23452 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786332336319544-28292-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786332336319544-28292-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786332336319544-28292-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:30:57.136142 23452 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:30:57.137065 23452 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:30:57.137367 23452 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:30:57.138510 23452 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:30:57.140125 23452 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:30:57.140228 23452 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:30:57.140336 23452 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:30:57.140393 23452 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:30:57.171846 23452 rpc_server.cc:307] RPC server started. Bound to: 127.27.161.2:45873
I20260810 03:30:57.171898 23645 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.161.2:45873 every 8 connection(s)
I20260810 03:30:57.173273 23452 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786332336319544-28292-0/minicluster-data/ts-1/data/info.pb
I20260810 03:30:57.180781 28292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iBH5/build/asan/bin/kudu as pid 23452
I20260810 03:30:57.181001 28292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786332336319544-28292-0/minicluster-data/ts-1/wal/instance
I20260810 03:30:57.181524 28292 external_mini_cluster.cc:1428] Setting key ee6e21e6a53f7bbe3962122339c1c10f
I20260810 03:30:57.184646 28292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iBH5/build/asan/bin/kudu
/tmp/dist-test-task24iBH5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786332336319544-28292-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786332336319544-28292-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786332336319544-28292-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786332336319544-28292-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.161.3:0
--local_ip_for_outbound_sockets=127.27.161.3
--webserver_interface=127.27.161.3
--webserver_port=0
--tserver_master_addrs=127.27.161.62:33913
--builtin_ntp_servers=127.27.161.20:38321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260810 03:30:57.186929 23448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:30:57.176040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.161.2:39441 (local address 127.27.161.62:33913)
0810 03:30:57.176271 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0810 03:30:57.176277 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:30:57.177655 (+ 1378us) server_negotiation.cc:407] Connection header received
0810 03:30:57.179487 (+ 1832us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:30:57.179492 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:30:57.179589 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:30:57.179682 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:30:57.180930 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:30:57.181836 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:30:57.183797 (+ 1961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:30:57.184221 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:30:57.184330 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:30:57.186073 (+ 1743us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:30:57.186079 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:30:57.186090 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:30:57.186111 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:30:57.186148 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:30:57.186162 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:30:57.186237 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0810 03:30:57.186572 (+ 335us) server_negotiation.cc:299] Negotiation successful
0810 03:30:57.186674 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260810 03:30:57.188377 23652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:30:57.176564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:33913 (local address 127.27.161.2:39441)
0810 03:30:57.177491 (+ 927us) negotiation.cc:107] Waiting for socket to connect
0810 03:30:57.177536 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0810 03:30:57.179214 (+ 1678us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:30:57.179880 (+ 666us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:30:57.179887 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:30:57.179981 (+ 94us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:30:57.180718 (+ 737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:30:57.180744 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:30:57.182031 (+ 1287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:30:57.182037 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:30:57.183614 (+ 1577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:30:57.183631 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:30:57.184467 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:30:57.184476 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:30:57.184811 (+ 335us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:30:57.185604 (+ 793us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 03:30:57.185608 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:30:57.185671 (+ 63us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 03:30:57.185796 (+ 125us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:30:57.186261 (+ 465us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:30:57.186273 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:30:57.186284 (+ 11us) client_negotiation.cc:769] Sending connection context
0810 03:30:57.186503 (+ 219us) client_negotiation.cc:240] Negotiation successful
0810 03:30:57.186973 (+ 470us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":281,"threads_started":1}
I20260810 03:30:57.190001 23646 heartbeater.cc:347] Connected to a master server at 127.27.161.62:33913
I20260810 03:30:57.190411 23646 heartbeater.cc:464] Registering TS with master...
I20260810 03:30:57.191468 23646 heartbeater.cc:511] Master 127.27.161.62:33913 requested a full tablet report, sending...
I20260810 03:30:57.193017 23167 ts_manager.cc:194] Registered new tserver with Master: e7c5365c5e98448eba0ba5683f057cea (127.27.161.2:45873)
I20260810 03:30:57.193871 23167 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.161.2:39441
W20260810 03:30:57.548467 23653 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:30:57.548799 23653 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:30:57.548931 23653 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:30:57.556378 23653 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:30:57.556501 23653 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:30:57.556650 23653 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.161.3
I20260810 03:30:57.566011 23653 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.161.20:38321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786332336319544-28292-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786332336319544-28292-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.161.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786332336319544-28292-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.161.3
--webserver_port=0
--tserver_master_addrs=127.27.161.62:33913
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.161.3
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786332336319544-28292-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:30:57.567438 23653 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:30:57.569370 23653 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:30:57.582589 23661 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:30:57.582605 23662 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:30:57.582809 23664 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:30:57.583976 23653 server_base.cc:1034] running on GCE node
I20260810 03:30:57.584609 23653 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:30:57.585608 23653 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:30:57.586809 23653 hybrid_clock.cc:648] HybridClock initialized: now 1786332657586756 us; error 45 us; skew 500 ppm
I20260810 03:30:57.587236 23653 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 03:30:57.589636 23653 webserver.cc:460] Webserver started at http://127.27.161.3:45769/ using document root <none> and password file <none>
I20260810 03:30:57.590435 23653 fs_manager.cc:359] Metadata directory not provided
I20260810 03:30:57.590548 23653 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:30:57.590920 23653 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:30:57.593935 23653 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786332336319544-28292-0/minicluster-data/ts-2/data/instance:
uuid: "fa178e7c694d44a8a8bbbc5bb794727c"
format_stamp: "Formatted at 2026-08-10 03:30:57 on dist-test-slave-vpvm"
server_key: "feae773ea93f3f61b0ad8e57af3996b2"
server_key_iv: "4ff1d387e7dd7f18fa11800106deb83b"
server_key_version: "encryptionkey@0"
I20260810 03:30:57.594815 23653 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786332336319544-28292-0/minicluster-data/ts-2/wal/instance:
uuid: "fa178e7c694d44a8a8bbbc5bb794727c"
format_stamp: "Formatted at 2026-08-10 03:30:57 on dist-test-slave-vpvm"
server_key: "feae773ea93f3f61b0ad8e57af3996b2"
server_key_iv: "4ff1d387e7dd7f18fa11800106deb83b"
server_key_version: "encryptionkey@0"
I20260810 03:30:57.601359 23653 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.002s
I20260810 03:30:57.605021 23674 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:30:57.606418 23653 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260810 03:30:57.606583 23653 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786332336319544-28292-0/minicluster-data/ts-2/data,/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786332336319544-28292-0/minicluster-data/ts-2/wal
uuid: "fa178e7c694d44a8a8bbbc5bb794727c"
format_stamp: "Formatted at 2026-08-10 03:30:57 on dist-test-slave-vpvm"
server_key: "feae773ea93f3f61b0ad8e57af3996b2"
server_key_iv: "4ff1d387e7dd7f18fa11800106deb83b"
server_key_version: "encryptionkey@0"
I20260810 03:30:57.606799 23653 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786332336319544-28292-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786332336319544-28292-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786332336319544-28292-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:30:57.624667 23653 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:30:57.625523 23653 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:30:57.625819 23653 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:30:57.626973 23653 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:30:57.628541 23653 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:30:57.628643 23653 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:30:57.628756 23653 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:30:57.628825 23653 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:30:57.662432 23653 rpc_server.cc:307] RPC server started. Bound to: 127.27.161.3:38641
I20260810 03:30:57.662490 23840 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.161.3:38641 every 8 connection(s)
I20260810 03:30:57.663842 23653 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786332336319544-28292-0/minicluster-data/ts-2/data/info.pb
I20260810 03:30:57.671482 28292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iBH5/build/asan/bin/kudu as pid 23653
I20260810 03:30:57.671690 28292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786332336319544-28292-0/minicluster-data/ts-2/wal/instance
I20260810 03:30:57.672144 28292 external_mini_cluster.cc:1428] Setting key d4845d148315154b9a87a47d8513bc98
I20260810 03:30:57.677536 23448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:30:57.666357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.161.3:43165 (local address 127.27.161.62:33913)
0810 03:30:57.666564 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0810 03:30:57.666569 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:30:57.667962 (+ 1393us) server_negotiation.cc:407] Connection header received
0810 03:30:57.669893 (+ 1931us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:30:57.669900 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:30:57.670039 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:30:57.670144 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:30:57.671387 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:30:57.672309 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:30:57.674441 (+ 2132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:30:57.675074 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:30:57.675216 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:30:57.676703 (+ 1487us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:30:57.676708 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:30:57.676712 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:30:57.676732 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:30:57.676767 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:30:57.676780 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:30:57.676859 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0810 03:30:57.677168 (+ 309us) server_negotiation.cc:299] Negotiation successful
0810 03:30:57.677269 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260810 03:30:57.678661 23844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:30:57.666859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:33913 (local address 127.27.161.3:43165)
0810 03:30:57.667787 (+ 928us) negotiation.cc:107] Waiting for socket to connect
0810 03:30:57.667835 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0810 03:30:57.669578 (+ 1743us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:30:57.670416 (+ 838us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:30:57.670425 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:30:57.670523 (+ 98us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:30:57.671175 (+ 652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:30:57.671195 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:30:57.672487 (+ 1292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:30:57.672491 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:30:57.674250 (+ 1759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:30:57.674267 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:30:57.675314 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:30:57.675320 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:30:57.675615 (+ 295us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:30:57.676327 (+ 712us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 03:30:57.676329 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:30:57.676376 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 03:30:57.676468 (+ 92us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:30:57.676890 (+ 422us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:30:57.676900 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:30:57.676908 (+ 8us) client_negotiation.cc:769] Sending connection context
0810 03:30:57.677079 (+ 171us) client_negotiation.cc:240] Negotiation successful
0810 03:30:57.677429 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":281,"threads_started":1}
I20260810 03:30:57.680182 23841 heartbeater.cc:347] Connected to a master server at 127.27.161.62:33913
I20260810 03:30:57.680509 23841 heartbeater.cc:464] Registering TS with master...
I20260810 03:30:57.681348 23841 heartbeater.cc:511] Master 127.27.161.62:33913 requested a full tablet report, sending...
I20260810 03:30:57.682951 23167 ts_manager.cc:194] Registered new tserver with Master: fa178e7c694d44a8a8bbbc5bb794727c (127.27.161.3:38641)
I20260810 03:30:57.683795 23167 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.161.3:43165
I20260810 03:30:57.687826 28292 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 03:30:57.698071 23448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:30:57.692077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43972 (local address 127.27.161.62:33913)
0810 03:30:57.692271 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0810 03:30:57.692277 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:30:57.692709 (+ 432us) server_negotiation.cc:407] Connection header received
0810 03:30:57.692891 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:30:57.692895 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:30:57.692965 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:30:57.693051 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:30:57.693605 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:30:57.694379 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:30:57.695772 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:30:57.696224 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:30:57.696336 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:30:57.697355 (+ 1019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:30:57.697360 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:30:57.697363 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:30:57.697382 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:30:57.697416 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:30:57.697428 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:30:57.697513 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0810 03:30:57.697635 (+ 122us) server_negotiation.cc:299] Negotiation successful
0810 03:30:57.697733 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260810 03:30:57.701903 23167 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43972:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 03:30:57.708817 23167 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 03:30:57.717433 23446 heartbeater.cc:503] Master 127.27.161.62:33913 was elected leader, sending a full tablet report...
I20260810 03:30:57.744656 23860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:30:57.722621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47918 (local address 127.27.161.2:45873)
0810 03:30:57.736373 (+ 13752us) server_negotiation.cc:206] Beginning negotiation
0810 03:30:57.736381 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 03:30:57.736420 (+ 39us) server_negotiation.cc:407] Connection header received
0810 03:30:57.736531 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:30:57.736541 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:30:57.736845 (+ 304us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:30:57.737083 (+ 238us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:30:57.738286 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:30:57.739367 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:30:57.742168 (+ 2801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:30:57.743471 (+ 1303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:30:57.743599 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:30:57.743868 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0810 03:30:57.744169 (+ 301us) server_negotiation.cc:299] Negotiation successful
0810 03:30:57.744349 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13587,"thread_start_us":13480,"threads_started":1}
I20260810 03:30:57.744679 23863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:30:57.722546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.2:45873 (local address 127.0.0.1:47918)
0810 03:30:57.732231 (+ 9685us) negotiation.cc:107] Waiting for socket to connect
0810 03:30:57.732277 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0810 03:30:57.732513 (+ 236us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:30:57.737421 (+ 4908us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:30:57.737430 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:30:57.737463 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:30:57.738086 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:30:57.738100 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:30:57.739515 (+ 1415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:30:57.739520 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:30:57.741916 (+ 2396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:30:57.741931 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:30:57.743687 (+ 1756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:30:57.743694 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:30:57.743914 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:30:57.743969 (+ 55us) client_negotiation.cc:769] Sending connection context
0810 03:30:57.744075 (+ 106us) client_negotiation.cc:240] Negotiation successful
0810 03:30:57.744285 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9380,"mutex_wait_us":4,"thread_start_us":16697,"threads_started":1}
I20260810 03:30:57.746753 23869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:30:57.722590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.3:38641 (local address 127.0.0.1:54066)
0810 03:30:57.736017 (+ 13427us) negotiation.cc:107] Waiting for socket to connect
0810 03:30:57.736040 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0810 03:30:57.736213 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:30:57.737503 (+ 1290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:30:57.737509 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:30:57.737547 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:30:57.738106 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:30:57.738120 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:30:57.740266 (+ 2146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:30:57.740274 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:30:57.742708 (+ 2434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:30:57.742725 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:30:57.744857 (+ 2132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:30:57.744862 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:30:57.745028 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:30:57.745061 (+ 33us) client_negotiation.cc:769] Sending connection context
0810 03:30:57.745147 (+ 86us) client_negotiation.cc:240] Negotiation successful
0810 03:30:57.745279 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13280,"mutex_wait_us":25,"thread_start_us":9180,"threads_started":1}
I20260810 03:30:57.747164 23867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:30:57.723386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54066 (local address 127.27.161.3:38641)
0810 03:30:57.735672 (+ 12286us) server_negotiation.cc:206] Beginning negotiation
0810 03:30:57.735682 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 03:30:57.736890 (+ 1208us) server_negotiation.cc:407] Connection header received
0810 03:30:57.737012 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:30:57.737020 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:30:57.737268 (+ 248us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:30:57.738324 (+ 1056us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:30:57.738637 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:30:57.740124 (+ 1487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:30:57.742916 (+ 2792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:30:57.744326 (+ 1410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:30:57.744449 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:30:57.744723 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0810 03:30:57.746627 (+ 1904us) server_negotiation.cc:299] Negotiation successful
0810 03:30:57.746803 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12091,"thread_start_us":11915,"threads_started":1}
I20260810 03:30:57.747856 23864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:30:57.724188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.1:44423 (local address 127.0.0.1:38710)
0810 03:30:57.739062 (+ 14874us) negotiation.cc:107] Waiting for socket to connect
0810 03:30:57.739081 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 03:30:57.739210 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:30:57.739845 (+ 635us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:30:57.739849 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:30:57.739882 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:30:57.740296 (+ 414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:30:57.740308 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:30:57.742232 (+ 1924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:30:57.742235 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:30:57.744151 (+ 1916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:30:57.744168 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:30:57.747125 (+ 2957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:30:57.747133 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:30:57.747301 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:30:57.747336 (+ 35us) client_negotiation.cc:769] Sending connection context
0810 03:30:57.747428 (+ 92us) client_negotiation.cc:240] Negotiation successful
0810 03:30:57.747557 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14753,"thread_start_us":11504,"threads_started":1}
I20260810 03:30:57.748669 23872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:30:57.724534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38710 (local address 127.27.161.1:44423)
0810 03:30:57.736794 (+ 12260us) server_negotiation.cc:206] Beginning negotiation
0810 03:30:57.736813 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0810 03:30:57.739203 (+ 2390us) server_negotiation.cc:407] Connection header received
0810 03:30:57.739407 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:30:57.739414 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:30:57.739638 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:30:57.739826 (+ 188us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:30:57.740591 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:30:57.742098 (+ 1507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:30:57.744391 (+ 2293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:30:57.746080 (+ 1689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:30:57.746225 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:30:57.746491 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0810 03:30:57.748125 (+ 1634us) server_negotiation.cc:299] Negotiation successful
0810 03:30:57.748318 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12055,"thread_start_us":12767,"threads_started":1}
I20260810 03:30:57.749310 23539 tablet_service.cc:1467] Processing CreateTablet for tablet a8761d424a0e41eeb87497e186aa6919 (DEFAULT_TABLE table=test-table [id=04b2205dfa3849b1ad68c60f43e9994d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:30:57.750906 23539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8761d424a0e41eeb87497e186aa6919. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:30:57.750653 23751 tablet_service.cc:1467] Processing CreateTablet for tablet a8761d424a0e41eeb87497e186aa6919 (DEFAULT_TABLE table=test-table [id=04b2205dfa3849b1ad68c60f43e9994d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:30:57.752019 23751 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8761d424a0e41eeb87497e186aa6919. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:30:57.752239 23350 tablet_service.cc:1467] Processing CreateTablet for tablet a8761d424a0e41eeb87497e186aa6919 (DEFAULT_TABLE table=test-table [id=04b2205dfa3849b1ad68c60f43e9994d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:30:57.753674 23350 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8761d424a0e41eeb87497e186aa6919. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:30:57.763200 23873 tablet_bootstrap.cc:492] T a8761d424a0e41eeb87497e186aa6919 P fa178e7c694d44a8a8bbbc5bb794727c: Bootstrap starting.
I20260810 03:30:57.766176 23874 tablet_bootstrap.cc:492] T a8761d424a0e41eeb87497e186aa6919 P e7c5365c5e98448eba0ba5683f057cea: Bootstrap starting.
I20260810 03:30:57.766975 23873 tablet_bootstrap.cc:654] T a8761d424a0e41eeb87497e186aa6919 P fa178e7c694d44a8a8bbbc5bb794727c: Neither blocks nor log segments found. Creating new log.
I20260810 03:30:57.768358 23873 log.cc:824] T a8761d424a0e41eeb87497e186aa6919 P fa178e7c694d44a8a8bbbc5bb794727c: Log is configured to *not* fsync() on all Append() calls
I20260810 03:30:57.770362 23874 tablet_bootstrap.cc:654] T a8761d424a0e41eeb87497e186aa6919 P e7c5365c5e98448eba0ba5683f057cea: Neither blocks nor log segments found. Creating new log.
I20260810 03:30:57.770428 23875 tablet_bootstrap.cc:492] T a8761d424a0e41eeb87497e186aa6919 P 2414144a3e51404ebbbad2733efe8c29: Bootstrap starting.
I20260810 03:30:57.771796 23874 log.cc:824] T a8761d424a0e41eeb87497e186aa6919 P e7c5365c5e98448eba0ba5683f057cea: Log is configured to *not* fsync() on all Append() calls
I20260810 03:30:57.774094 23875 tablet_bootstrap.cc:654] T a8761d424a0e41eeb87497e186aa6919 P 2414144a3e51404ebbbad2733efe8c29: Neither blocks nor log segments found. Creating new log.
I20260810 03:30:57.774646 23873 tablet_bootstrap.cc:492] T a8761d424a0e41eeb87497e186aa6919 P fa178e7c694d44a8a8bbbc5bb794727c: No bootstrap required, opened a new log
I20260810 03:30:57.774919 23873 ts_tablet_manager.cc:1397] T a8761d424a0e41eeb87497e186aa6919 P fa178e7c694d44a8a8bbbc5bb794727c: Time spent bootstrapping tablet: real 0.012s user 0.010s sys 0.000s
I20260810 03:30:57.775031 23874 tablet_bootstrap.cc:492] T a8761d424a0e41eeb87497e186aa6919 P e7c5365c5e98448eba0ba5683f057cea: No bootstrap required, opened a new log
I20260810 03:30:57.775323 23874 ts_tablet_manager.cc:1397] T a8761d424a0e41eeb87497e186aa6919 P e7c5365c5e98448eba0ba5683f057cea: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260810 03:30:57.775405 23875 log.cc:824] T a8761d424a0e41eeb87497e186aa6919 P 2414144a3e51404ebbbad2733efe8c29: Log is configured to *not* fsync() on all Append() calls
I20260810 03:30:57.779983 23875 tablet_bootstrap.cc:492] T a8761d424a0e41eeb87497e186aa6919 P 2414144a3e51404ebbbad2733efe8c29: No bootstrap required, opened a new log
I20260810 03:30:57.780296 23875 ts_tablet_manager.cc:1397] T a8761d424a0e41eeb87497e186aa6919 P 2414144a3e51404ebbbad2733efe8c29: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.004s
I20260810 03:30:57.783123 23873 raft_consensus.cc:348] T a8761d424a0e41eeb87497e186aa6919 P fa178e7c694d44a8a8bbbc5bb794727c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7c5365c5e98448eba0ba5683f057cea" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 45873 } } peers { permanent_uuid: "fa178e7c694d44a8a8bbbc5bb794727c" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 38641 } } peers { permanent_uuid: "2414144a3e51404ebbbad2733efe8c29" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 44423 } }
I20260810 03:30:57.783564 23873 raft_consensus.cc:374] T a8761d424a0e41eeb87497e186aa6919 P fa178e7c694d44a8a8bbbc5bb794727c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:30:57.783679 23873 raft_consensus.cc:729] T a8761d424a0e41eeb87497e186aa6919 P fa178e7c694d44a8a8bbbc5bb794727c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa178e7c694d44a8a8bbbc5bb794727c, State: Initialized, Role: FOLLOWER
I20260810 03:30:57.784300 23873 consensus_queue.cc:260] T a8761d424a0e41eeb87497e186aa6919 P fa178e7c694d44a8a8bbbc5bb794727c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7c5365c5e98448eba0ba5683f057cea" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 45873 } } peers { permanent_uuid: "fa178e7c694d44a8a8bbbc5bb794727c" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 38641 } } peers { permanent_uuid: "2414144a3e51404ebbbad2733efe8c29" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 44423 } }
I20260810 03:30:57.786706 23874 raft_consensus.cc:348] T a8761d424a0e41eeb87497e186aa6919 P e7c5365c5e98448eba0ba5683f057cea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7c5365c5e98448eba0ba5683f057cea" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 45873 } } peers { permanent_uuid: "fa178e7c694d44a8a8bbbc5bb794727c" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 38641 } } peers { permanent_uuid: "2414144a3e51404ebbbad2733efe8c29" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 44423 } }
I20260810 03:30:57.787192 23874 raft_consensus.cc:374] T a8761d424a0e41eeb87497e186aa6919 P e7c5365c5e98448eba0ba5683f057cea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:30:57.787226 23841 heartbeater.cc:503] Master 127.27.161.62:33913 was elected leader, sending a full tablet report...
I20260810 03:30:57.787297 23874 raft_consensus.cc:729] T a8761d424a0e41eeb87497e186aa6919 P e7c5365c5e98448eba0ba5683f057cea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7c5365c5e98448eba0ba5683f057cea, State: Initialized, Role: FOLLOWER
I20260810 03:30:57.787961 23874 consensus_queue.cc:260] T a8761d424a0e41eeb87497e186aa6919 P e7c5365c5e98448eba0ba5683f057cea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7c5365c5e98448eba0ba5683f057cea" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 45873 } } peers { permanent_uuid: "fa178e7c694d44a8a8bbbc5bb794727c" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 38641 } } peers { permanent_uuid: "2414144a3e51404ebbbad2733efe8c29" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 44423 } }
I20260810 03:30:57.789382 23873 ts_tablet_manager.cc:1428] T a8761d424a0e41eeb87497e186aa6919 P fa178e7c694d44a8a8bbbc5bb794727c: Time spent starting tablet: real 0.014s user 0.009s sys 0.004s
I20260810 03:30:57.790705 23646 heartbeater.cc:503] Master 127.27.161.62:33913 was elected leader, sending a full tablet report...
I20260810 03:30:57.790798 23875 raft_consensus.cc:348] T a8761d424a0e41eeb87497e186aa6919 P 2414144a3e51404ebbbad2733efe8c29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7c5365c5e98448eba0ba5683f057cea" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 45873 } } peers { permanent_uuid: "fa178e7c694d44a8a8bbbc5bb794727c" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 38641 } } peers { permanent_uuid: "2414144a3e51404ebbbad2733efe8c29" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 44423 } }
I20260810 03:30:57.791275 23875 raft_consensus.cc:374] T a8761d424a0e41eeb87497e186aa6919 P 2414144a3e51404ebbbad2733efe8c29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:30:57.791376 23875 raft_consensus.cc:729] T a8761d424a0e41eeb87497e186aa6919 P 2414144a3e51404ebbbad2733efe8c29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2414144a3e51404ebbbad2733efe8c29, State: Initialized, Role: FOLLOWER
I20260810 03:30:57.791463 23874 ts_tablet_manager.cc:1428] T a8761d424a0e41eeb87497e186aa6919 P e7c5365c5e98448eba0ba5683f057cea: Time spent starting tablet: real 0.016s user 0.015s sys 0.000s
I20260810 03:30:57.792092 23875 consensus_queue.cc:260] T a8761d424a0e41eeb87497e186aa6919 P 2414144a3e51404ebbbad2733efe8c29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7c5365c5e98448eba0ba5683f057cea" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 45873 } } peers { permanent_uuid: "fa178e7c694d44a8a8bbbc5bb794727c" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 38641 } } peers { permanent_uuid: "2414144a3e51404ebbbad2733efe8c29" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 44423 } }
I20260810 03:30:57.795080 23875 ts_tablet_manager.cc:1428] T a8761d424a0e41eeb87497e186aa6919 P 2414144a3e51404ebbbad2733efe8c29: Time spent starting tablet: real 0.015s user 0.015s sys 0.001s
I20260810 03:30:57.906713 23888 raft_consensus.cc:482] T a8761d424a0e41eeb87497e186aa6919 P 2414144a3e51404ebbbad2733efe8c29 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 03:30:57.907090 23888 raft_consensus.cc:504] T a8761d424a0e41eeb87497e186aa6919 P 2414144a3e51404ebbbad2733efe8c29 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7c5365c5e98448eba0ba5683f057cea" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 45873 } } peers { permanent_uuid: "fa178e7c694d44a8a8bbbc5bb794727c" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 38641 } } peers { permanent_uuid: "2414144a3e51404ebbbad2733efe8c29" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 44423 } }
I20260810 03:30:57.909070 23888 leader_election.cc:288] T a8761d424a0e41eeb87497e186aa6919 P 2414144a3e51404ebbbad2733efe8c29 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e7c5365c5e98448eba0ba5683f057cea (127.27.161.2:45873), fa178e7c694d44a8a8bbbc5bb794727c (127.27.161.3:38641)
W20260810 03:30:57.915571 23842 tablet.cc:2369] T a8761d424a0e41eeb87497e186aa6919 P fa178e7c694d44a8a8bbbc5bb794727c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 03:30:57.915865 23867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:30:57.909795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.161.1:49529 (local address 127.27.161.3:38641)
0810 03:30:57.910019 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0810 03:30:57.910026 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:30:57.910501 (+ 475us) server_negotiation.cc:407] Connection header received
0810 03:30:57.910669 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:30:57.910673 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:30:57.910745 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:30:57.910822 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:30:57.911402 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:30:57.912147 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:30:57.914085 (+ 1938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:30:57.914892 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:30:57.915056 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:30:57.915133 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0810 03:30:57.915466 (+ 333us) server_negotiation.cc:299] Negotiation successful
0810 03:30:57.915613 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260810 03:30:57.916121 23860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:30:57.909527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.161.1:34671 (local address 127.27.161.2:45873)
0810 03:30:57.909763 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0810 03:30:57.909769 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:30:57.910591 (+ 822us) server_negotiation.cc:407] Connection header received
0810 03:30:57.910793 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:30:57.910798 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:30:57.910900 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:30:57.911005 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:30:57.911682 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:30:57.912414 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:30:57.914347 (+ 1933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:30:57.915148 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:30:57.915310 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:30:57.915387 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0810 03:30:57.915694 (+ 307us) server_negotiation.cc:299] Negotiation successful
0810 03:30:57.915830 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260810 03:30:57.916296 23890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:30:57.909478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.2:45873 (local address 127.27.161.1:34671)
0810 03:30:57.910445 (+ 967us) negotiation.cc:107] Waiting for socket to connect
0810 03:30:57.910483 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0810 03:30:57.910646 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:30:57.911046 (+ 400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:30:57.911051 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:30:57.911082 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:30:57.911505 (+ 423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:30:57.911515 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:30:57.912570 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:30:57.912576 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:30:57.914146 (+ 1570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:30:57.914160 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:30:57.915361 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:30:57.915369 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:30:57.915526 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:30:57.915553 (+ 27us) client_negotiation.cc:769] Sending connection context
0810 03:30:57.915645 (+ 92us) client_negotiation.cc:240] Negotiation successful
0810 03:30:57.915784 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":827,"mutex_wait_us":24,"thread_start_us":576,"threads_started":1}
I20260810 03:30:57.916868 23889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:30:57.909403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.3:38641 (local address 127.27.161.1:49529)
0810 03:30:57.910354 (+ 951us) negotiation.cc:107] Waiting for socket to connect
0810 03:30:57.910373 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 03:30:57.910533 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:30:57.910871 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:30:57.910875 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:30:57.910899 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:30:57.911258 (+ 359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:30:57.911268 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:30:57.912289 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:30:57.912293 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:30:57.913875 (+ 1582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:30:57.913889 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:30:57.915085 (+ 1196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:30:57.915092 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:30:57.915258 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:30:57.915293 (+ 35us) client_negotiation.cc:769] Sending connection context
0810 03:30:57.916404 (+ 1111us) client_negotiation.cc:240] Negotiation successful
0810 03:30:57.916542 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":791,"mutex_wait_us":25,"thread_start_us":567,"threads_started":1}
I20260810 03:30:57.917448 23572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8761d424a0e41eeb87497e186aa6919" candidate_uuid: "2414144a3e51404ebbbad2733efe8c29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7c5365c5e98448eba0ba5683f057cea" is_pre_election: true
I20260810 03:30:57.917912 23782 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8761d424a0e41eeb87497e186aa6919" candidate_uuid: "2414144a3e51404ebbbad2733efe8c29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa178e7c694d44a8a8bbbc5bb794727c" is_pre_election: true
I20260810 03:30:57.918129 23572 raft_consensus.cc:2454] T a8761d424a0e41eeb87497e186aa6919 P e7c5365c5e98448eba0ba5683f057cea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2414144a3e51404ebbbad2733efe8c29 in term 0.
I20260810 03:30:57.918490 23782 raft_consensus.cc:2454] T a8761d424a0e41eeb87497e186aa6919 P fa178e7c694d44a8a8bbbc5bb794727c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2414144a3e51404ebbbad2733efe8c29 in term 0.
I20260810 03:30:57.918921 23282 leader_election.cc:302] T a8761d424a0e41eeb87497e186aa6919 P 2414144a3e51404ebbbad2733efe8c29 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2414144a3e51404ebbbad2733efe8c29, e7c5365c5e98448eba0ba5683f057cea; no voters:
I20260810 03:30:57.919313 23888 raft_consensus.cc:2781] T a8761d424a0e41eeb87497e186aa6919 P 2414144a3e51404ebbbad2733efe8c29 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 03:30:57.919466 23888 raft_consensus.cc:482] T a8761d424a0e41eeb87497e186aa6919 P 2414144a3e51404ebbbad2733efe8c29 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 03:30:57.919575 23888 raft_consensus.cc:3037] T a8761d424a0e41eeb87497e186aa6919 P 2414144a3e51404ebbbad2733efe8c29 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:30:57.921780 23888 raft_consensus.cc:504] T a8761d424a0e41eeb87497e186aa6919 P 2414144a3e51404ebbbad2733efe8c29 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7c5365c5e98448eba0ba5683f057cea" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 45873 } } peers { permanent_uuid: "fa178e7c694d44a8a8bbbc5bb794727c" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 38641 } } peers { permanent_uuid: "2414144a3e51404ebbbad2733efe8c29" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 44423 } }
I20260810 03:30:57.922581 23888 leader_election.cc:288] T a8761d424a0e41eeb87497e186aa6919 P 2414144a3e51404ebbbad2733efe8c29 [CANDIDATE]: Term 1 election: Requested vote from peers e7c5365c5e98448eba0ba5683f057cea (127.27.161.2:45873), fa178e7c694d44a8a8bbbc5bb794727c (127.27.161.3:38641)
I20260810 03:30:57.923157 23782 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8761d424a0e41eeb87497e186aa6919" candidate_uuid: "2414144a3e51404ebbbad2733efe8c29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa178e7c694d44a8a8bbbc5bb794727c"
I20260810 03:30:57.923166 23572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8761d424a0e41eeb87497e186aa6919" candidate_uuid: "2414144a3e51404ebbbad2733efe8c29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7c5365c5e98448eba0ba5683f057cea"
I20260810 03:30:57.923398 23782 raft_consensus.cc:3037] T a8761d424a0e41eeb87497e186aa6919 P fa178e7c694d44a8a8bbbc5bb794727c [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:30:57.923398 23572 raft_consensus.cc:3037] T a8761d424a0e41eeb87497e186aa6919 P e7c5365c5e98448eba0ba5683f057cea [term 0 FOLLOWER]: Advancing to term 1
W20260810 03:30:57.925276 23648 tablet.cc:2369] T a8761d424a0e41eeb87497e186aa6919 P e7c5365c5e98448eba0ba5683f057cea: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 03:30:57.926682 23572 raft_consensus.cc:2454] T a8761d424a0e41eeb87497e186aa6919 P e7c5365c5e98448eba0ba5683f057cea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2414144a3e51404ebbbad2733efe8c29 in term 1.
I20260810 03:30:57.926682 23782 raft_consensus.cc:2454] T a8761d424a0e41eeb87497e186aa6919 P fa178e7c694d44a8a8bbbc5bb794727c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2414144a3e51404ebbbad2733efe8c29 in term 1.
I20260810 03:30:57.927209 23282 leader_election.cc:302] T a8761d424a0e41eeb87497e186aa6919 P 2414144a3e51404ebbbad2733efe8c29 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 2414144a3e51404ebbbad2733efe8c29, e7c5365c5e98448eba0ba5683f057cea, fa178e7c694d44a8a8bbbc5bb794727c; no voters:
I20260810 03:30:57.927578 23888 raft_consensus.cc:2781] T a8761d424a0e41eeb87497e186aa6919 P 2414144a3e51404ebbbad2733efe8c29 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:30:57.928050 23888 raft_consensus.cc:686] T a8761d424a0e41eeb87497e186aa6919 P 2414144a3e51404ebbbad2733efe8c29 [term 1 LEADER]: Becoming Leader. State: Replica: 2414144a3e51404ebbbad2733efe8c29, State: Running, Role: LEADER
I20260810 03:30:57.928627 23888 consensus_queue.cc:237] T a8761d424a0e41eeb87497e186aa6919 P 2414144a3e51404ebbbad2733efe8c29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7c5365c5e98448eba0ba5683f057cea" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 45873 } } peers { permanent_uuid: "fa178e7c694d44a8a8bbbc5bb794727c" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 38641 } } peers { permanent_uuid: "2414144a3e51404ebbbad2733efe8c29" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 44423 } }
I20260810 03:30:57.933320 23167 catalog_manager.cc:5495] T a8761d424a0e41eeb87497e186aa6919 P 2414144a3e51404ebbbad2733efe8c29 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2414144a3e51404ebbbad2733efe8c29 (127.27.161.1). New cstate: current_term: 1 leader_uuid: "2414144a3e51404ebbbad2733efe8c29" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7c5365c5e98448eba0ba5683f057cea" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 45873 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fa178e7c694d44a8a8bbbc5bb794727c" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 38641 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2414144a3e51404ebbbad2733efe8c29" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 44423 } health_report { overall_health: HEALTHY } } }
W20260810 03:30:57.944788 23447 tablet.cc:2369] T a8761d424a0e41eeb87497e186aa6919 P 2414144a3e51404ebbbad2733efe8c29: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 03:30:57.949574 23872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:30:57.944046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38714 (local address 127.27.161.1:44423)
0810 03:30:57.944265 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0810 03:30:57.944270 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:30:57.944289 (+ 19us) server_negotiation.cc:407] Connection header received
0810 03:30:57.944538 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:30:57.944544 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:30:57.944695 (+ 151us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:30:57.944868 (+ 173us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 03:30:57.945225 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:30:57.946114 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:30:57.947566 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:30:57.948006 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:30:57.948105 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:30:57.948563 (+ 458us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 03:30:57.949063 (+ 500us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 03:30:57.949172 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0810 03:30:57.949233 (+ 61us) server_negotiation.cc:299] Negotiation successful
0810 03:30:57.949303 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260810 03:30:57.958630 23572 raft_consensus.cc:1261] T a8761d424a0e41eeb87497e186aa6919 P e7c5365c5e98448eba0ba5683f057cea [term 1 FOLLOWER]: Refusing update from remote peer 2414144a3e51404ebbbad2733efe8c29: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:30:57.958640 23782 raft_consensus.cc:1261] T a8761d424a0e41eeb87497e186aa6919 P fa178e7c694d44a8a8bbbc5bb794727c [term 1 FOLLOWER]: Refusing update from remote peer 2414144a3e51404ebbbad2733efe8c29: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:30:57.959663 23888 consensus_queue.cc:1035] T a8761d424a0e41eeb87497e186aa6919 P 2414144a3e51404ebbbad2733efe8c29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7c5365c5e98448eba0ba5683f057cea" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 45873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:30:57.959995 23891 consensus_queue.cc:1035] T a8761d424a0e41eeb87497e186aa6919 P 2414144a3e51404ebbbad2733efe8c29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa178e7c694d44a8a8bbbc5bb794727c" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 38641 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:30:57.993848 23165 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:43972:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 03:30:57.994163 23165 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:43972:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 03:30:57.998407 23165 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 37d955abfde248809ce4ee9e5e39fe79: Sending DeleteTablet for 3 replicas of tablet a8761d424a0e41eeb87497e186aa6919
I20260810 03:30:57.999851 23539 tablet_service.cc:1514] Processing DeleteTablet for tablet a8761d424a0e41eeb87497e186aa6919 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:30:57 UTC) from {username='slave'} at 127.0.0.1:47918
I20260810 03:30:57.999851 23751 tablet_service.cc:1514] Processing DeleteTablet for tablet a8761d424a0e41eeb87497e186aa6919 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:30:57 UTC) from {username='slave'} at 127.0.0.1:54066
I20260810 03:30:58.000169 23350 tablet_service.cc:1514] Processing DeleteTablet for tablet a8761d424a0e41eeb87497e186aa6919 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:30:57 UTC) from {username='slave'} at 127.0.0.1:38710
I20260810 03:30:58.002156 28292 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iBH5/build/asan/bin/kudu pid 23260
I20260810 03:30:58.018680 23907 tablet_replica.cc:330] T a8761d424a0e41eeb87497e186aa6919 P fa178e7c694d44a8a8bbbc5bb794727c: stopping tablet replica
I20260810 03:30:58.019109 23908 tablet_replica.cc:330] T a8761d424a0e41eeb87497e186aa6919 P e7c5365c5e98448eba0ba5683f057cea: stopping tablet replica
I20260810 03:30:58.019768 23908 raft_consensus.cc:2229] T a8761d424a0e41eeb87497e186aa6919 P e7c5365c5e98448eba0ba5683f057cea [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 03:30:58.020427 23908 raft_consensus.cc:2258] T a8761d424a0e41eeb87497e186aa6919 P e7c5365c5e98448eba0ba5683f057cea [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:30:58.020607 23907 raft_consensus.cc:2229] T a8761d424a0e41eeb87497e186aa6919 P fa178e7c694d44a8a8bbbc5bb794727c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 03:30:58.021006 23910 tablet_replica.cc:330] T a8761d424a0e41eeb87497e186aa6919 P 2414144a3e51404ebbbad2733efe8c29: stopping tablet replica
I20260810 03:30:58.022807 23907 raft_consensus.cc:2258] T a8761d424a0e41eeb87497e186aa6919 P fa178e7c694d44a8a8bbbc5bb794727c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:30:58.023399 23910 raft_consensus.cc:2229] T a8761d424a0e41eeb87497e186aa6919 P 2414144a3e51404ebbbad2733efe8c29 [term 1 LEADER]: Raft consensus shutting down.
I20260810 03:30:58.024251 23910 raft_consensus.cc:2258] T a8761d424a0e41eeb87497e186aa6919 P 2414144a3e51404ebbbad2733efe8c29 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:30:58.024835 23908 ts_tablet_manager.cc:1905] T a8761d424a0e41eeb87497e186aa6919 P e7c5365c5e98448eba0ba5683f057cea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:30:58.025669 23872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:30:58.003684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38724 (local address 127.27.161.1:44423)
0810 03:30:58.003908 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0810 03:30:58.003916 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 03:30:58.016240 (+ 12324us) server_negotiation.cc:407] Connection header received
0810 03:30:58.016593 (+ 353us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:30:58.016599 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:30:58.016741 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:30:58.016839 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:30:58.017603 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:30:58.019487 (+ 1884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:30:58.021332 (+ 1845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:30:58.022031 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:30:58.022147 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:30:58.023713 (+ 1566us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:30:58.023728 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:30:58.023748 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:30:58.024632 (+ 884us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:30:58.024697 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:30:58.024729 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:30:58.024815 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0810 03:30:58.025129 (+ 314us) server_negotiation.cc:299] Negotiation successful
0810 03:30:58.025272 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260810 03:30:58.027228 23436 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:30:58.027645 23907 ts_tablet_manager.cc:1905] T a8761d424a0e41eeb87497e186aa6919 P fa178e7c694d44a8a8bbbc5bb794727c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:30:58.033161 23908 ts_tablet_manager.cc:1918] T a8761d424a0e41eeb87497e186aa6919 P e7c5365c5e98448eba0ba5683f057cea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 03:30:58.033335 23908 log.cc:1196] T a8761d424a0e41eeb87497e186aa6919 P e7c5365c5e98448eba0ba5683f057cea: Deleting WAL directory at /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786332336319544-28292-0/minicluster-data/ts-1/wal/wals/a8761d424a0e41eeb87497e186aa6919
I20260810 03:30:58.033798 23908 ts_tablet_manager.cc:1939] T a8761d424a0e41eeb87497e186aa6919 P e7c5365c5e98448eba0ba5683f057cea: Deleting consensus metadata
I20260810 03:30:58.035522 23907 ts_tablet_manager.cc:1918] T a8761d424a0e41eeb87497e186aa6919 P fa178e7c694d44a8a8bbbc5bb794727c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 03:30:58.035570 23147 catalog_manager.cc:4841] TS e7c5365c5e98448eba0ba5683f057cea (127.27.161.2:45873): tablet a8761d424a0e41eeb87497e186aa6919 (table test-table [id=04b2205dfa3849b1ad68c60f43e9994d]) successfully deleted
I20260810 03:30:58.035706 23907 log.cc:1196] T a8761d424a0e41eeb87497e186aa6919 P fa178e7c694d44a8a8bbbc5bb794727c: Deleting WAL directory at /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786332336319544-28292-0/minicluster-data/ts-2/wal/wals/a8761d424a0e41eeb87497e186aa6919
I20260810 03:30:58.036242 23907 ts_tablet_manager.cc:1939] T a8761d424a0e41eeb87497e186aa6919 P fa178e7c694d44a8a8bbbc5bb794727c: Deleting consensus metadata
I20260810 03:30:58.038188 23143 catalog_manager.cc:4841] TS fa178e7c694d44a8a8bbbc5bb794727c (127.27.161.3:38641): tablet a8761d424a0e41eeb87497e186aa6919 (table test-table [id=04b2205dfa3849b1ad68c60f43e9994d]) successfully deleted
I20260810 03:30:58.027652 23910 ts_tablet_manager.cc:1905] T a8761d424a0e41eeb87497e186aa6919 P 2414144a3e51404ebbbad2733efe8c29: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:30:58.127995 28292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iBH5/build/asan/bin/kudu with pid 23260
W20260810 03:30:58.153751 23146 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.161.1:44423 (error 108)
I20260810 03:30:58.153837 28292 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iBH5/build/asan/bin/kudu pid 23452
I20260810 03:30:58.155951 23864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:30:58.155372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.1:44423 (local address 127.0.0.1:38730)
0810 03:30:58.155597 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0810 03:30:58.155763 (+ 166us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.1:44423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
W20260810 03:30:58.156257 23146 catalog_manager.cc:4584] TS 2414144a3e51404ebbbad2733efe8c29 (127.27.161.1:44423): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a8761d424a0e41eeb87497e186aa6919: Network error: Client connection negotiation failed: client connection to 127.27.161.1:44423: connect: Connection refused (error 111)
I20260810 03:30:58.161310 23860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:30:58.154783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47934 (local address 127.27.161.2:45873)
0810 03:30:58.154986 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0810 03:30:58.154992 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:30:58.155012 (+ 20us) server_negotiation.cc:407] Connection header received
0810 03:30:58.155129 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:30:58.155135 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:30:58.155251 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:30:58.155380 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:30:58.155919 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:30:58.156938 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:30:58.158401 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:30:58.158834 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:30:58.158936 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:30:58.159969 (+ 1033us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:30:58.159978 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:30:58.159994 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:30:58.160603 (+ 609us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:30:58.160653 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:30:58.160674 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:30:58.160766 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0810 03:30:58.160878 (+ 112us) server_negotiation.cc:299] Negotiation successful
0810 03:30:58.160976 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260810 03:30:58.162067 23639 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:30:58.210242 23864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:30:58.209586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.1:44423 (local address 127.0.0.1:38732)
0810 03:30:58.209853 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0810 03:30:58.210026 (+ 173us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.1:44423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260810 03:30:58.211494 23864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:30:58.211040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.1:44423 (local address 127.0.0.1:38746)
0810 03:30:58.211230 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0810 03:30:58.211330 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.1:44423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260810 03:30:58.256707 28292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iBH5/build/asan/bin/kudu with pid 23452
I20260810 03:30:58.272208 23864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:30:58.271742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.1:44423 (local address 127.0.0.1:38758)
0810 03:30:58.271961 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0810 03:30:58.272054 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.1:44423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260810 03:30:58.273339 23864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:30:58.272975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.1:44423 (local address 127.0.0.1:38774)
0810 03:30:58.273125 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0810 03:30:58.273202 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.1:44423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260810 03:30:58.282007 28292 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iBH5/build/asan/bin/kudu pid 23653
I20260810 03:30:58.289188 23867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:30:58.282855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54068 (local address 127.27.161.3:38641)
0810 03:30:58.283029 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0810 03:30:58.283033 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:30:58.283074 (+ 41us) server_negotiation.cc:407] Connection header received
0810 03:30:58.283264 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:30:58.283268 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:30:58.283366 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:30:58.283456 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:30:58.284069 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:30:58.284851 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:30:58.286357 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:30:58.286809 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:30:58.286919 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:30:58.287962 (+ 1043us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:30:58.287971 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:30:58.287987 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:30:58.288567 (+ 580us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:30:58.288616 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:30:58.288638 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:30:58.288738 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0810 03:30:58.288828 (+ 90us) server_negotiation.cc:299] Negotiation successful
0810 03:30:58.288936 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260810 03:30:58.289932 23835 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:30:58.354058 23864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:30:58.353485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.1:44423 (local address 127.0.0.1:38780)
0810 03:30:58.353725 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0810 03:30:58.353817 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.1:44423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
I20260810 03:30:58.355140 23864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:30:58.354742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.1:44423 (local address 127.0.0.1:38786)
0810 03:30:58.354943 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0810 03:30:58.355026 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.1:44423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
I20260810 03:30:58.380466 28292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iBH5/build/asan/bin/kudu with pid 23653
I20260810 03:30:58.403525 28292 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iBH5/build/asan/bin/kudu pid 23116
I20260810 03:30:58.404621 23218 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:30:58.515914 28292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iBH5/build/asan/bin/kudu with pid 23116
2026-08-10T03:30:58Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (2998 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786332336319544-28292-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 03:30:58 dist-test-slave-vpvm krb5kdc[23928](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 03:30:58 dist-test-slave-vpvm krb5kdc[23928](info): set up 2 sockets
Aug 10 03:30:58 dist-test-slave-vpvm krb5kdc[23928](info): commencing operation
krb5kdc: starting...
W20260810 03:30:59.760550 28292 mini_kdc.cc:121] Time spent starting KDC: real 1.198s user 0.006s sys 0.048s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786332336319544-28292-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786332336319544-28292-0/krb5kdc/test-user.keytab.
Aug 10 03:30:59 dist-test-slave-vpvm krb5kdc[23928](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332659, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T03:31:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T03:31:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.161.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.161.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.161.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.62.keytab.
Entry for principal kudu/127.27.161.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.62.keytab.
I20260810 03:31:00.109337 28292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iBH5/build/asan/bin/kudu
/tmp/dist-test-task24iBH5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786332336319544-28292-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786332336319544-28292-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786332336319544-28292-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786332336319544-28292-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.161.62:45507
--webserver_interface=127.27.161.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.62.keytab
--principal=kudu/127.27.161.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.161.20:39381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.161.62:45507
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786332336319544-28292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786332336319544-28292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786332336319544-28292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:31:00.472517 23950 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:31:00.472853 23950 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:31:00.472993 23950 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:31:00.480523 23950 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:31:00.480664 23950 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:31:00.480719 23950 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:31:00.480800 23950 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:31:00.480857 23950 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:31:00.491704 23950 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.161.20:39381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786332336319544-28292-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786332336319544-28292-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.161.62:45507
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.62.keytab
--principal=kudu/127.27.161.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.161.62:45507
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786332336319544-28292-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.161.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786332336319544-28292-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:31:00.493184 23950 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:31:00.494642 23950 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:31:00.508199 23958 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:31:00.508148 23959 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:31:00.508378 23961 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:31:00.509088 23950 server_base.cc:1034] running on GCE node
I20260810 03:31:00.509999 23950 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:31:00.511571 23950 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:31:00.512790 23950 hybrid_clock.cc:648] HybridClock initialized: now 1786332660512728 us; error 52 us; skew 500 ppm
I20260810 03:31:00.513172 23950 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:31:00 dist-test-slave-vpvm krb5kdc[23928](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332660, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.161.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:31:00.516304 23950 init.cc:377] Logged in from keytab as kudu/127.27.161.62@KRBTEST.COM (short username kudu)
I20260810 03:31:00.517889 23950 webserver.cc:460] Webserver started at http://127.27.161.62:46873/ using document root <none> and password file <none>
I20260810 03:31:00.518657 23950 fs_manager.cc:359] Metadata directory not provided
I20260810 03:31:00.518785 23950 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:31:00.519109 23950 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:31:00.522344 23950 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786332336319544-28292-0/minicluster-data/master-0/data/instance:
uuid: "0ec4d8ba88a94fd8838397056fac4bc4"
format_stamp: "Formatted at 2026-08-10 03:31:00 on dist-test-slave-vpvm"
server_key: "8c43deee3cea214faab0a44feab0c5dd"
server_key_iv: "8dde91047388120b4cf0da670f91bc41"
server_key_version: "encryptionkey@0"
I20260810 03:31:00.523159 23950 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786332336319544-28292-0/minicluster-data/master-0/wal/instance:
uuid: "0ec4d8ba88a94fd8838397056fac4bc4"
format_stamp: "Formatted at 2026-08-10 03:31:00 on dist-test-slave-vpvm"
server_key: "8c43deee3cea214faab0a44feab0c5dd"
server_key_iv: "8dde91047388120b4cf0da670f91bc41"
server_key_version: "encryptionkey@0"
I20260810 03:31:00.528898 23950 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260810 03:31:00.532810 23969 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:31:00.534332 23950 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 03:31:00.534515 23950 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786332336319544-28292-0/minicluster-data/master-0/data,/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786332336319544-28292-0/minicluster-data/master-0/wal
uuid: "0ec4d8ba88a94fd8838397056fac4bc4"
format_stamp: "Formatted at 2026-08-10 03:31:00 on dist-test-slave-vpvm"
server_key: "8c43deee3cea214faab0a44feab0c5dd"
server_key_iv: "8dde91047388120b4cf0da670f91bc41"
server_key_version: "encryptionkey@0"
I20260810 03:31:00.534719 23950 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786332336319544-28292-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786332336319544-28292-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786332336319544-28292-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:31:00.559836 23950 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:31:00.564123 23950 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:31:00.564407 23950 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:31:00.583405 23950 rpc_server.cc:307] RPC server started. Bound to: 127.27.161.62:45507
I20260810 03:31:00.583406 24050 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.161.62:45507 every 8 connection(s)
I20260810 03:31:00.584921 23950 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786332336319544-28292-0/minicluster-data/master-0/data/info.pb
I20260810 03:31:00.589262 24051 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:31:00.594877 28292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iBH5/build/asan/bin/kudu as pid 23950
I20260810 03:31:00.595058 28292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786332336319544-28292-0/minicluster-data/master-0/wal/instance
I20260810 03:31:00.595507 28292 external_mini_cluster.cc:1428] Setting key a669f4c416c00b65809a8e65c09aeff7
I20260810 03:31:00.603547 24051 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ec4d8ba88a94fd8838397056fac4bc4: Bootstrap starting.
I20260810 03:31:00.607852 24051 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0ec4d8ba88a94fd8838397056fac4bc4: Neither blocks nor log segments found. Creating new log.
I20260810 03:31:00.609328 24051 log.cc:824] T 00000000000000000000000000000000 P 0ec4d8ba88a94fd8838397056fac4bc4: Log is configured to *not* fsync() on all Append() calls
I20260810 03:31:00.612772 24051 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ec4d8ba88a94fd8838397056fac4bc4: No bootstrap required, opened a new log
Aug 10 03:31:00 dist-test-slave-vpvm krb5kdc[23928](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332659, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.161.62@KRBTEST.COM
I20260810 03:31:00.622466 24056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:31:00.597586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33648 (local address 127.27.161.62:45507)
0810 03:31:00.598602 (+ 1016us) server_negotiation.cc:206] Beginning negotiation
0810 03:31:00.598621 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0810 03:31:00.598690 (+ 69us) server_negotiation.cc:407] Connection header received
0810 03:31:00.601459 (+ 2769us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:31:00.601511 (+ 52us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:31:00.602188 (+ 677us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:31:00.603086 (+ 898us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:31:00.604603 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:31:00.606770 (+ 2167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:31:00.608364 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:31:00.609505 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:31:00.609729 (+ 224us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:31:00.614439 (+ 4710us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:31:00.614449 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:31:00.614466 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:31:00.614509 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:31:00.617836 (+ 3327us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:31:00.618820 (+ 984us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:31:00.618828 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:31:00.618836 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:31:00.619042 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:31:00.619482 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:31:00.619486 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:31:00.619488 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:31:00.620107 (+ 619us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:31:00.620368 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0810 03:31:00.620847 (+ 479us) server_negotiation.cc:299] Negotiation successful
0810 03:31:00.621271 (+ 424us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":690,"thread_start_us":403,"threads_started":1}
I20260810 03:31:00.622440 24051 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0ec4d8ba88a94fd8838397056fac4bc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ec4d8ba88a94fd8838397056fac4bc4" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 45507 } }
I20260810 03:31:00.622977 24051 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0ec4d8ba88a94fd8838397056fac4bc4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:31:00.623109 24051 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0ec4d8ba88a94fd8838397056fac4bc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ec4d8ba88a94fd8838397056fac4bc4, State: Initialized, Role: FOLLOWER
I20260810 03:31:00.623835 24051 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0ec4d8ba88a94fd8838397056fac4bc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ec4d8ba88a94fd8838397056fac4bc4" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 45507 } }
I20260810 03:31:00.624135 24051 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0ec4d8ba88a94fd8838397056fac4bc4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 03:31:00.624293 24051 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0ec4d8ba88a94fd8838397056fac4bc4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 03:31:00.624466 24051 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0ec4d8ba88a94fd8838397056fac4bc4 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:31:00.627197 24051 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0ec4d8ba88a94fd8838397056fac4bc4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ec4d8ba88a94fd8838397056fac4bc4" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 45507 } }
I20260810 03:31:00.627768 24051 leader_election.cc:302] T 00000000000000000000000000000000 P 0ec4d8ba88a94fd8838397056fac4bc4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0ec4d8ba88a94fd8838397056fac4bc4; no voters:
I20260810 03:31:00.628364 24051 leader_election.cc:288] T 00000000000000000000000000000000 P 0ec4d8ba88a94fd8838397056fac4bc4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 03:31:00.628511 24059 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0ec4d8ba88a94fd8838397056fac4bc4 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:31:00.629597 24059 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0ec4d8ba88a94fd8838397056fac4bc4 [term 1 LEADER]: Becoming Leader. State: Replica: 0ec4d8ba88a94fd8838397056fac4bc4, State: Running, Role: LEADER
I20260810 03:31:00.630237 24059 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0ec4d8ba88a94fd8838397056fac4bc4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ec4d8ba88a94fd8838397056fac4bc4" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 45507 } }
I20260810 03:31:00.630771 24051 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0ec4d8ba88a94fd8838397056fac4bc4 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:31:00.633091 24060 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ec4d8ba88a94fd8838397056fac4bc4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0ec4d8ba88a94fd8838397056fac4bc4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ec4d8ba88a94fd8838397056fac4bc4" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 45507 } } }
I20260810 03:31:00.633188 24061 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ec4d8ba88a94fd8838397056fac4bc4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0ec4d8ba88a94fd8838397056fac4bc4. Latest consensus state: current_term: 1 leader_uuid: "0ec4d8ba88a94fd8838397056fac4bc4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ec4d8ba88a94fd8838397056fac4bc4" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 45507 } } }
I20260810 03:31:00.633517 24060 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ec4d8ba88a94fd8838397056fac4bc4 [sys.catalog]: This master's current role is: LEADER
I20260810 03:31:00.633668 24061 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ec4d8ba88a94fd8838397056fac4bc4 [sys.catalog]: This master's current role is: LEADER
I20260810 03:31:00.639536 24067 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 03:31:00.645414 24067 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 03:31:00.655941 24067 catalog_manager.cc:1347] Generated new cluster ID: bca2c651d5c94a4cb7cb1ea1aefa03fc
I20260810 03:31:00.656082 24067 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 03:31:00.667574 24067 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 03:31:00.669612 24067 catalog_manager.cc:1504] Loading token signing keys...
I20260810 03:31:00.677253 24067 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0ec4d8ba88a94fd8838397056fac4bc4: Generated new TSK 0
I20260810 03:31:00.678449 24067 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.161.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.161.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.161.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.1.keytab.
Entry for principal kudu/127.27.161.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.1.keytab.
I20260810 03:31:00.773001 28292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iBH5/build/asan/bin/kudu
/tmp/dist-test-task24iBH5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786332336319544-28292-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786332336319544-28292-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786332336319544-28292-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786332336319544-28292-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.161.1:0
--local_ip_for_outbound_sockets=127.27.161.1
--webserver_interface=127.27.161.1
--webserver_port=0
--tserver_master_addrs=127.27.161.62:45507
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.1.keytab
--principal=kudu/127.27.161.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.161.20:39381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786332336319544-28292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786332336319544-28292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786332336319544-28292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:31:01.139987 24087 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:31:01.140300 24087 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:31:01.140399 24087 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:31:01.148313 24087 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:31:01.148450 24087 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:31:01.148571 24087 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.161.1
I20260810 03:31:01.159478 24087 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.161.20:39381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786332336319544-28292-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786332336319544-28292-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.1.keytab
--principal=kudu/127.27.161.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.161.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786332336319544-28292-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.161.1
--webserver_port=0
--tserver_master_addrs=127.27.161.62:45507
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.161.1
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786332336319544-28292-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:31:01.160847 24087 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:31:01.162304 24087 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:31:01.175709 24096 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:31:01.175639 24098 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:31:01.175741 24100 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:31:01.176771 24087 server_base.cc:1034] running on GCE node
I20260810 03:31:01.177412 24087 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:31:01.178491 24087 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:31:01.179700 24087 hybrid_clock.cc:648] HybridClock initialized: now 1786332661179648 us; error 44 us; skew 500 ppm
I20260810 03:31:01.180045 24087 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:31:01 dist-test-slave-vpvm krb5kdc[23928](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332661, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.161.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:31:01.187496 24087 init.cc:377] Logged in from keytab as kudu/127.27.161.1@KRBTEST.COM (short username kudu)
I20260810 03:31:01.189404 24087 webserver.cc:460] Webserver started at http://127.27.161.1:44627/ using document root <none> and password file <none>
I20260810 03:31:01.190287 24087 fs_manager.cc:359] Metadata directory not provided
I20260810 03:31:01.190445 24087 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:31:01.190806 24087 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:31:01.194324 24087 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786332336319544-28292-0/minicluster-data/ts-0/data/instance:
uuid: "a6279bfc96804d3798e8aa48b17a84d5"
format_stamp: "Formatted at 2026-08-10 03:31:01 on dist-test-slave-vpvm"
server_key: "ccf66029fa1fc173ce907646441f8ad0"
server_key_iv: "7e5eea2ed6a9b954cf1cf1389c2384fb"
server_key_version: "encryptionkey@0"
I20260810 03:31:01.195230 24087 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786332336319544-28292-0/minicluster-data/ts-0/wal/instance:
uuid: "a6279bfc96804d3798e8aa48b17a84d5"
format_stamp: "Formatted at 2026-08-10 03:31:01 on dist-test-slave-vpvm"
server_key: "ccf66029fa1fc173ce907646441f8ad0"
server_key_iv: "7e5eea2ed6a9b954cf1cf1389c2384fb"
server_key_version: "encryptionkey@0"
I20260810 03:31:01.201327 24087 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260810 03:31:01.205574 24112 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:31:01.207182 24087 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 03:31:01.207386 24087 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786332336319544-28292-0/minicluster-data/ts-0/data,/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786332336319544-28292-0/minicluster-data/ts-0/wal
uuid: "a6279bfc96804d3798e8aa48b17a84d5"
format_stamp: "Formatted at 2026-08-10 03:31:01 on dist-test-slave-vpvm"
server_key: "ccf66029fa1fc173ce907646441f8ad0"
server_key_iv: "7e5eea2ed6a9b954cf1cf1389c2384fb"
server_key_version: "encryptionkey@0"
I20260810 03:31:01.207612 24087 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786332336319544-28292-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786332336319544-28292-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786332336319544-28292-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:31:01.225276 24087 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:31:01.230013 24087 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:31:01.230319 24087 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:31:01.231442 24087 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:31:01.232986 24087 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:31:01.233067 24087 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:31:01.233201 24087 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:31:01.233258 24087 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:31:01.266086 24087 rpc_server.cc:307] RPC server started. Bound to: 127.27.161.1:37673
I20260810 03:31:01.266120 24288 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.161.1:37673 every 8 connection(s)
I20260810 03:31:01.267416 24087 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786332336319544-28292-0/minicluster-data/ts-0/data/info.pb
I20260810 03:31:01.273772 28292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iBH5/build/asan/bin/kudu as pid 24087
I20260810 03:31:01.274021 28292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786332336319544-28292-0/minicluster-data/ts-0/wal/instance
I20260810 03:31:01.274457 28292 external_mini_cluster.cc:1428] Setting key e6dc4a03d035eb59e4ba5c6c6e35a0fa
Aug 10 03:31:01 dist-test-slave-vpvm krb5kdc[23928](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332661, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.161.1@KRBTEST.COM for kudu/127.27.161.62@KRBTEST.COM
I20260810 03:31:01.289994 24293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:31:01.270147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.161.1:43833 (local address 127.27.161.62:45507)
0810 03:31:01.270627 (+ 480us) server_negotiation.cc:206] Beginning negotiation
0810 03:31:01.270632 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:31:01.271680 (+ 1048us) server_negotiation.cc:407] Connection header received
0810 03:31:01.273609 (+ 1929us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:31:01.273615 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:31:01.273701 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:31:01.273799 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:31:01.275678 (+ 1879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:31:01.276584 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:31:01.278287 (+ 1703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:31:01.278869 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:31:01.279149 (+ 280us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:31:01.283168 (+ 4019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:31:01.283175 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:31:01.283180 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:31:01.283244 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:31:01.286462 (+ 3218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:31:01.287413 (+ 951us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:31:01.287420 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:31:01.287423 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:31:01.287582 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:31:01.288077 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:31:01.288081 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:31:01.288084 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:31:01.288459 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:31:01.288605 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0810 03:31:01.289358 (+ 753us) server_negotiation.cc:299] Negotiation successful
0810 03:31:01.289623 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":368,"thread_start_us":237,"threads_started":1}
I20260810 03:31:01.291121 24294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:31:01.270648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:45507 (local address 127.27.161.1:43833)
0810 03:31:01.271468 (+ 820us) negotiation.cc:107] Waiting for socket to connect
0810 03:31:01.271535 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0810 03:31:01.273276 (+ 1741us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:31:01.274124 (+ 848us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:31:01.274137 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:31:01.274755 (+ 618us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:31:01.275467 (+ 712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:31:01.275509 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:31:01.276779 (+ 1270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:31:01.276784 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:31:01.278111 (+ 1327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:31:01.278125 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:31:01.279260 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:31:01.279266 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:31:01.279503 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:31:01.280227 (+ 724us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:31:01.280255 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:31:01.282845 (+ 2590us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:31:01.286682 (+ 3837us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:31:01.286692 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:31:01.286709 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:31:01.287252 (+ 543us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:31:01.287716 (+ 464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:31:01.287719 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:31:01.287722 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:31:01.287959 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:31:01.288580 (+ 621us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:31:01.288587 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:31:01.288955 (+ 368us) client_negotiation.cc:769] Sending connection context
0810 03:31:01.289197 (+ 242us) client_negotiation.cc:240] Negotiation successful
0810 03:31:01.289794 (+ 597us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":528,"thread_start_us":305,"threads_started":1}
I20260810 03:31:01.292872 24291 heartbeater.cc:347] Connected to a master server at 127.27.161.62:45507
I20260810 03:31:01.293216 24291 heartbeater.cc:464] Registering TS with master...
I20260810 03:31:01.294049 24291 heartbeater.cc:511] Master 127.27.161.62:45507 requested a full tablet report, sending...
I20260810 03:31:01.296298 23997 ts_manager.cc:194] Registered new tserver with Master: a6279bfc96804d3798e8aa48b17a84d5 (127.27.161.1:37673)
I20260810 03:31:01.297820 23997 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.161.1@KRBTEST.COM'} at 127.27.161.1:43833
WARNING: no policy specified for kudu/127.27.161.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.161.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.161.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.2.keytab.
Entry for principal kudu/127.27.161.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.2.keytab.
I20260810 03:31:01.349136 28292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iBH5/build/asan/bin/kudu
/tmp/dist-test-task24iBH5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786332336319544-28292-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786332336319544-28292-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786332336319544-28292-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786332336319544-28292-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.161.2:0
--local_ip_for_outbound_sockets=127.27.161.2
--webserver_interface=127.27.161.2
--webserver_port=0
--tserver_master_addrs=127.27.161.62:45507
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.2.keytab
--principal=kudu/127.27.161.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.161.20:39381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786332336319544-28292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786332336319544-28292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786332336319544-28292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:31:01.717113 24299 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:31:01.717439 24299 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:31:01.717589 24299 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:31:01.725672 24299 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:31:01.725822 24299 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:31:01.726040 24299 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.161.2
I20260810 03:31:01.736301 24299 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.161.20:39381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786332336319544-28292-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786332336319544-28292-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.2.keytab
--principal=kudu/127.27.161.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.161.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786332336319544-28292-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.161.2
--webserver_port=0
--tserver_master_addrs=127.27.161.62:45507
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.161.2
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786332336319544-28292-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:31:01.737789 24299 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:31:01.739414 24299 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:31:01.753551 24307 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:31:01.753481 24314 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:31:01.753508 24308 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:31:01.754433 24299 server_base.cc:1034] running on GCE node
I20260810 03:31:01.755146 24299 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:31:01.756207 24299 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:31:01.757418 24299 hybrid_clock.cc:648] HybridClock initialized: now 1786332661757366 us; error 50 us; skew 500 ppm
I20260810 03:31:01.757754 24299 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:31:01 dist-test-slave-vpvm krb5kdc[23928](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332661, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.161.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:31:01.761037 24299 init.cc:377] Logged in from keytab as kudu/127.27.161.2@KRBTEST.COM (short username kudu)
I20260810 03:31:01.762718 24299 webserver.cc:460] Webserver started at http://127.27.161.2:41453/ using document root <none> and password file <none>
I20260810 03:31:01.763484 24299 fs_manager.cc:359] Metadata directory not provided
I20260810 03:31:01.763619 24299 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:31:01.763952 24299 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:31:01.767194 24299 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786332336319544-28292-0/minicluster-data/ts-1/data/instance:
uuid: "d896337cd18049b094acb8c47b3f6e18"
format_stamp: "Formatted at 2026-08-10 03:31:01 on dist-test-slave-vpvm"
server_key: "b36752ed7c44d8de9629a090589f835b"
server_key_iv: "aefcba80377d08d9998e558f024659e3"
server_key_version: "encryptionkey@0"
I20260810 03:31:01.768021 24299 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786332336319544-28292-0/minicluster-data/ts-1/wal/instance:
uuid: "d896337cd18049b094acb8c47b3f6e18"
format_stamp: "Formatted at 2026-08-10 03:31:01 on dist-test-slave-vpvm"
server_key: "b36752ed7c44d8de9629a090589f835b"
server_key_iv: "aefcba80377d08d9998e558f024659e3"
server_key_version: "encryptionkey@0"
I20260810 03:31:01.773530 24299 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260810 03:31:01.777133 24332 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:31:01.778529 24299 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 03:31:01.778713 24299 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786332336319544-28292-0/minicluster-data/ts-1/data,/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786332336319544-28292-0/minicluster-data/ts-1/wal
uuid: "d896337cd18049b094acb8c47b3f6e18"
format_stamp: "Formatted at 2026-08-10 03:31:01 on dist-test-slave-vpvm"
server_key: "b36752ed7c44d8de9629a090589f835b"
server_key_iv: "aefcba80377d08d9998e558f024659e3"
server_key_version: "encryptionkey@0"
I20260810 03:31:01.778903 24299 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786332336319544-28292-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786332336319544-28292-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786332336319544-28292-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:31:01.805267 24299 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:31:01.809677 24299 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:31:01.809984 24299 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:31:01.810988 24299 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:31:01.812426 24299 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:31:01.812505 24299 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:31:01.812629 24299 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:31:01.812688 24299 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:31:01.843686 24299 rpc_server.cc:307] RPC server started. Bound to: 127.27.161.2:42953
I20260810 03:31:01.843743 24502 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.161.2:42953 every 8 connection(s)
I20260810 03:31:01.845001 24299 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786332336319544-28292-0/minicluster-data/ts-1/data/info.pb
I20260810 03:31:01.854923 28292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iBH5/build/asan/bin/kudu as pid 24299
I20260810 03:31:01.855134 28292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786332336319544-28292-0/minicluster-data/ts-1/wal/instance
I20260810 03:31:01.855803 28292 external_mini_cluster.cc:1428] Setting key 994d78c7566ef2f4bc038aba72b5a971
Aug 10 03:31:01 dist-test-slave-vpvm krb5kdc[23928](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332661, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.161.2@KRBTEST.COM for kudu/127.27.161.62@KRBTEST.COM
I20260810 03:31:01.867120 24508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:31:01.847692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.161.2:53337 (local address 127.27.161.62:45507)
0810 03:31:01.848193 (+ 501us) server_negotiation.cc:206] Beginning negotiation
0810 03:31:01.848200 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:31:01.849128 (+ 928us) server_negotiation.cc:407] Connection header received
0810 03:31:01.851014 (+ 1886us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:31:01.851018 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:31:01.851092 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:31:01.851192 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:31:01.852788 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:31:01.853528 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:31:01.855739 (+ 2211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:31:01.856231 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:31:01.856345 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:31:01.860774 (+ 4429us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:31:01.860780 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:31:01.860783 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:31:01.860819 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:31:01.863654 (+ 2835us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:31:01.864539 (+ 885us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:31:01.864544 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:31:01.864547 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:31:01.864706 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:31:01.865170 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:31:01.865174 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:31:01.865177 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:31:01.865523 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:31:01.865672 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0810 03:31:01.866503 (+ 831us) server_negotiation.cc:299] Negotiation successful
0810 03:31:01.866744 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":365,"thread_start_us":237,"threads_started":1}
I20260810 03:31:01.868191 24509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:31:01.848205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:45507 (local address 127.27.161.2:53337)
0810 03:31:01.848951 (+ 746us) negotiation.cc:107] Waiting for socket to connect
0810 03:31:01.849000 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0810 03:31:01.850747 (+ 1747us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:31:01.851413 (+ 666us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:31:01.851421 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:31:01.851965 (+ 544us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:31:01.852614 (+ 649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:31:01.852634 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:31:01.853719 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:31:01.853724 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:31:01.855515 (+ 1791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:31:01.855543 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:31:01.856484 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:31:01.856490 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:31:01.856785 (+ 295us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:31:01.857542 (+ 757us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:31:01.857566 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:31:01.860494 (+ 2928us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:31:01.863824 (+ 3330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:31:01.863834 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:31:01.863852 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:31:01.864402 (+ 550us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:31:01.864800 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:31:01.864804 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:31:01.864807 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:31:01.865057 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:31:01.865634 (+ 577us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:31:01.865642 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:31:01.866106 (+ 464us) client_negotiation.cc:769] Sending connection context
0810 03:31:01.866381 (+ 275us) client_negotiation.cc:240] Negotiation successful
0810 03:31:01.866898 (+ 517us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":498,"thread_start_us":277,"threads_started":1}
I20260810 03:31:01.869921 24503 heartbeater.cc:347] Connected to a master server at 127.27.161.62:45507
I20260810 03:31:01.870313 24503 heartbeater.cc:464] Registering TS with master...
I20260810 03:31:01.871248 24503 heartbeater.cc:511] Master 127.27.161.62:45507 requested a full tablet report, sending...
I20260810 03:31:01.873003 23997 ts_manager.cc:194] Registered new tserver with Master: d896337cd18049b094acb8c47b3f6e18 (127.27.161.2:42953)
I20260810 03:31:01.873936 23997 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.161.2@KRBTEST.COM'} at 127.27.161.2:53337
WARNING: no policy specified for kudu/127.27.161.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.161.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.161.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.3.keytab.
Entry for principal kudu/127.27.161.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.3.keytab.
I20260810 03:31:01.929445 28292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iBH5/build/asan/bin/kudu
/tmp/dist-test-task24iBH5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786332336319544-28292-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786332336319544-28292-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786332336319544-28292-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786332336319544-28292-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.161.3:0
--local_ip_for_outbound_sockets=127.27.161.3
--webserver_interface=127.27.161.3
--webserver_port=0
--tserver_master_addrs=127.27.161.62:45507
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.3.keytab
--principal=kudu/127.27.161.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.161.20:39381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786332336319544-28292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786332336319544-28292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786332336319544-28292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:31:02.292723 24513 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:31:02.293058 24513 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:31:02.293210 24513 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:31:02.300789 24513 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:31:02.300951 24513 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:31:02.301121 24513 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.161.3
I20260810 03:31:02.301162 24291 heartbeater.cc:503] Master 127.27.161.62:45507 was elected leader, sending a full tablet report...
I20260810 03:31:02.312177 24513 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.161.20:39381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786332336319544-28292-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786332336319544-28292-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.3.keytab
--principal=kudu/127.27.161.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.161.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786332336319544-28292-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.161.3
--webserver_port=0
--tserver_master_addrs=127.27.161.62:45507
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.161.3
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786332336319544-28292-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:31:02.313925 24513 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:31:02.315398 24513 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:31:02.330435 24523 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:31:02.330480 24525 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:31:02.330538 24527 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:31:02.332024 24513 server_base.cc:1034] running on GCE node
I20260810 03:31:02.332747 24513 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:31:02.333868 24513 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:31:02.335116 24513 hybrid_clock.cc:648] HybridClock initialized: now 1786332662335062 us; error 47 us; skew 500 ppm
I20260810 03:31:02.335472 24513 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:31:02 dist-test-slave-vpvm krb5kdc[23928](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332662, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.161.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:31:02.339887 24513 init.cc:377] Logged in from keytab as kudu/127.27.161.3@KRBTEST.COM (short username kudu)
I20260810 03:31:02.341686 24513 webserver.cc:460] Webserver started at http://127.27.161.3:44001/ using document root <none> and password file <none>
I20260810 03:31:02.342499 24513 fs_manager.cc:359] Metadata directory not provided
I20260810 03:31:02.342643 24513 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:31:02.342986 24513 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:31:02.346158 24513 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786332336319544-28292-0/minicluster-data/ts-2/data/instance:
uuid: "506789d474ef48f1b4ec1d0839a3ca6c"
format_stamp: "Formatted at 2026-08-10 03:31:02 on dist-test-slave-vpvm"
server_key: "20066e1c60a340bd5bdb6c3c1011a25f"
server_key_iv: "5f0a02f83ba767dae116f9e54fb27dbe"
server_key_version: "encryptionkey@0"
I20260810 03:31:02.346979 24513 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786332336319544-28292-0/minicluster-data/ts-2/wal/instance:
uuid: "506789d474ef48f1b4ec1d0839a3ca6c"
format_stamp: "Formatted at 2026-08-10 03:31:02 on dist-test-slave-vpvm"
server_key: "20066e1c60a340bd5bdb6c3c1011a25f"
server_key_iv: "5f0a02f83ba767dae116f9e54fb27dbe"
server_key_version: "encryptionkey@0"
I20260810 03:31:02.352793 24513 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260810 03:31:02.356370 24537 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:31:02.357811 24513 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 03:31:02.358055 24513 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786332336319544-28292-0/minicluster-data/ts-2/data,/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786332336319544-28292-0/minicluster-data/ts-2/wal
uuid: "506789d474ef48f1b4ec1d0839a3ca6c"
format_stamp: "Formatted at 2026-08-10 03:31:02 on dist-test-slave-vpvm"
server_key: "20066e1c60a340bd5bdb6c3c1011a25f"
server_key_iv: "5f0a02f83ba767dae116f9e54fb27dbe"
server_key_version: "encryptionkey@0"
I20260810 03:31:02.358265 24513 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786332336319544-28292-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786332336319544-28292-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786332336319544-28292-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:31:02.378144 24513 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:31:02.382144 24513 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:31:02.382483 24513 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:31:02.383589 24513 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:31:02.385015 24513 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:31:02.385097 24513 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:31:02.385229 24513 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:31:02.385281 24513 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:31:02.418599 24513 rpc_server.cc:307] RPC server started. Bound to: 127.27.161.3:42921
I20260810 03:31:02.418648 24706 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.161.3:42921 every 8 connection(s)
I20260810 03:31:02.419934 24513 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786332336319544-28292-0/minicluster-data/ts-2/data/info.pb
I20260810 03:31:02.425783 28292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iBH5/build/asan/bin/kudu as pid 24513
I20260810 03:31:02.426026 28292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786332336319544-28292-0/minicluster-data/ts-2/wal/instance
I20260810 03:31:02.426638 28292 external_mini_cluster.cc:1428] Setting key 0a2c44364a896a9771f146163a3b8875
Aug 10 03:31:02 dist-test-slave-vpvm krb5kdc[23928](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332662, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.161.3@KRBTEST.COM for kudu/127.27.161.62@KRBTEST.COM
I20260810 03:31:02.441111 24709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:31:02.422739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.161.3:56797 (local address 127.27.161.62:45507)
0810 03:31:02.423316 (+ 577us) server_negotiation.cc:206] Beginning negotiation
0810 03:31:02.423321 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:31:02.424434 (+ 1113us) server_negotiation.cc:407] Connection header received
0810 03:31:02.426511 (+ 2077us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:31:02.426517 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:31:02.426608 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:31:02.426710 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:31:02.428691 (+ 1981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:31:02.429573 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:31:02.431249 (+ 1676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:31:02.431718 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:31:02.431831 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:31:02.435334 (+ 3503us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:31:02.435340 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:31:02.435343 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:31:02.435381 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:31:02.437902 (+ 2521us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:31:02.438704 (+ 802us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:31:02.438710 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:31:02.438713 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:31:02.438851 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:31:02.439281 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:31:02.439285 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:31:02.439287 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:31:02.439564 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:31:02.439714 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0810 03:31:02.440474 (+ 760us) server_negotiation.cc:299] Negotiation successful
0810 03:31:02.440718 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":387,"thread_start_us":249,"threads_started":1}
I20260810 03:31:02.442353 24711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:31:02.423437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:45507 (local address 127.27.161.3:56797)
0810 03:31:02.424257 (+ 820us) negotiation.cc:107] Waiting for socket to connect
0810 03:31:02.424296 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0810 03:31:02.426211 (+ 1915us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:31:02.426992 (+ 781us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:31:02.427000 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:31:02.427708 (+ 708us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:31:02.428497 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:31:02.428528 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:31:02.429737 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:31:02.429742 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:31:02.431104 (+ 1362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:31:02.431118 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:31:02.431960 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:31:02.431969 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:31:02.432210 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:31:02.432831 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:31:02.432860 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:31:02.435111 (+ 2251us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:31:02.438055 (+ 2944us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:31:02.438066 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:31:02.438085 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:31:02.438581 (+ 496us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:31:02.438957 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:31:02.438963 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:31:02.438966 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:31:02.439162 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:31:02.439699 (+ 537us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:31:02.439707 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:31:02.440067 (+ 360us) client_negotiation.cc:769] Sending connection context
0810 03:31:02.440337 (+ 270us) client_negotiation.cc:240] Negotiation successful
0810 03:31:02.440931 (+ 594us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":556,"thread_start_us":328,"threads_started":1}
I20260810 03:31:02.444198 24707 heartbeater.cc:347] Connected to a master server at 127.27.161.62:45507
I20260810 03:31:02.444502 24707 heartbeater.cc:464] Registering TS with master...
I20260810 03:31:02.445252 24707 heartbeater.cc:511] Master 127.27.161.62:45507 requested a full tablet report, sending...
I20260810 03:31:02.446795 23997 ts_manager.cc:194] Registered new tserver with Master: 506789d474ef48f1b4ec1d0839a3ca6c (127.27.161.3:42921)
I20260810 03:31:02.447659 23997 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.161.3@KRBTEST.COM'} at 127.27.161.3:56797
I20260810 03:31:02.454941 28292 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 03:31:02.472138 24709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:31:02.459127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33652 (local address 127.27.161.62:45507)
0810 03:31:02.459349 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0810 03:31:02.459358 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 03:31:02.459736 (+ 378us) server_negotiation.cc:407] Connection header received
0810 03:31:02.459897 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:31:02.459901 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:31:02.459975 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:31:02.460059 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:31:02.461620 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:31:02.462443 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:31:02.463832 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:31:02.464264 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:31:02.464360 (+ 96us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:31:02.467222 (+ 2862us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:31:02.467227 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:31:02.467231 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:31:02.467272 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:31:02.469629 (+ 2357us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:31:02.470422 (+ 793us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:31:02.470426 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:31:02.470428 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:31:02.470549 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:31:02.470884 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:31:02.470886 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:31:02.470888 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:31:02.471176 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:31:02.471379 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0810 03:31:02.471690 (+ 311us) server_negotiation.cc:299] Negotiation successful
0810 03:31:02.471833 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260810 03:31:02.476025 23997 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33652:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 03:31:02.483094 23997 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 03:31:02.518388 24726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:31:02.496165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.1:37673 (local address 127.0.0.1:33942)
0810 03:31:02.508483 (+ 12318us) negotiation.cc:107] Waiting for socket to connect
0810 03:31:02.508527 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0810 03:31:02.508795 (+ 268us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:31:02.509630 (+ 835us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:31:02.509639 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:31:02.509680 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:31:02.510244 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:31:02.510265 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:31:02.512398 (+ 2133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:31:02.512404 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:31:02.514888 (+ 2484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:31:02.514906 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:31:02.517225 (+ 2319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:31:02.517232 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:31:02.517463 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:31:02.517530 (+ 67us) client_negotiation.cc:769] Sending connection context
0810 03:31:02.517653 (+ 123us) client_negotiation.cc:240] Negotiation successful
0810 03:31:02.517872 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12032,"mutex_wait_us":5,"thread_start_us":13375,"threads_started":1}
I20260810 03:31:02.518893 24725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:31:02.496285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.2:42953 (local address 127.0.0.1:44024)
0810 03:31:02.509876 (+ 13591us) negotiation.cc:107] Waiting for socket to connect
0810 03:31:02.509895 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 03:31:02.510071 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:31:02.511220 (+ 1149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:31:02.511225 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:31:02.511253 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:31:02.511761 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:31:02.511776 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:31:02.513709 (+ 1933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:31:02.513715 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:31:02.515869 (+ 2154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:31:02.515886 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:31:02.518024 (+ 2138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:31:02.518030 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:31:02.518215 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:31:02.518248 (+ 33us) client_negotiation.cc:769] Sending connection context
0810 03:31:02.518351 (+ 103us) client_negotiation.cc:240] Negotiation successful
0810 03:31:02.518486 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13446,"mutex_wait_us":37,"thread_start_us":11756,"threads_started":1}
I20260810 03:31:02.519235 24729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:31:02.496720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33942 (local address 127.27.161.1:37673)
0810 03:31:02.507576 (+ 10856us) server_negotiation.cc:206] Beginning negotiation
0810 03:31:02.507588 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 03:31:02.508886 (+ 1298us) server_negotiation.cc:407] Connection header received
0810 03:31:02.509013 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:31:02.509021 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:31:02.509323 (+ 302us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:31:02.509551 (+ 228us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:31:02.510464 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:31:02.512229 (+ 1765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:31:02.515124 (+ 2895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:31:02.516658 (+ 1534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:31:02.516798 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:31:02.517067 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0810 03:31:02.518682 (+ 1615us) server_negotiation.cc:299] Negotiation successful
0810 03:31:02.518884 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10579,"thread_start_us":10889,"threads_started":1}
I20260810 03:31:02.519843 24727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:31:02.496651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44024 (local address 127.27.161.2:42953)
0810 03:31:02.509462 (+ 12811us) server_negotiation.cc:206] Beginning negotiation
0810 03:31:02.509472 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 03:31:02.510290 (+ 818us) server_negotiation.cc:407] Connection header received
0810 03:31:02.510402 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:31:02.510410 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:31:02.510742 (+ 332us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:31:02.510942 (+ 200us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:31:02.511963 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:31:02.513522 (+ 1559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:31:02.516065 (+ 2543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:31:02.517482 (+ 1417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:31:02.517649 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:31:02.517835 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0810 03:31:02.519313 (+ 1478us) server_negotiation.cc:299] Negotiation successful
0810 03:31:02.519506 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12634,"thread_start_us":12487,"threads_started":1}
I20260810 03:31:02.523746 24403 tablet_service.cc:1467] Processing CreateTablet for tablet 663072df88a04261b12dd903bc7e1fbe (DEFAULT_TABLE table=test-table [id=54cd887aaf734023a332d33c3bfeebeb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:31:02.523751 24188 tablet_service.cc:1467] Processing CreateTablet for tablet 663072df88a04261b12dd903bc7e1fbe (DEFAULT_TABLE table=test-table [id=54cd887aaf734023a332d33c3bfeebeb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:31:02.525453 24403 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 663072df88a04261b12dd903bc7e1fbe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:31:02.525452 24188 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 663072df88a04261b12dd903bc7e1fbe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:31:02.530036 24731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:31:02.510471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33652 (local address 127.27.161.3:42921)
0810 03:31:02.521006 (+ 10535us) server_negotiation.cc:206] Beginning negotiation
0810 03:31:02.521016 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 03:31:02.521055 (+ 39us) server_negotiation.cc:407] Connection header received
0810 03:31:02.521163 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:31:02.521173 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:31:02.521615 (+ 442us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:31:02.521865 (+ 250us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:31:02.522867 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:31:02.525086 (+ 2219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:31:02.527817 (+ 2731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:31:02.528830 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:31:02.528978 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:31:02.529259 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0810 03:31:02.529475 (+ 216us) server_negotiation.cc:299] Negotiation successful
0810 03:31:02.529654 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10331,"thread_start_us":9827,"threads_started":1}
I20260810 03:31:02.530732 24730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:31:02.510124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.3:42921 (local address 127.0.0.1:33652)
0810 03:31:02.516769 (+ 6645us) negotiation.cc:107] Waiting for socket to connect
0810 03:31:02.516794 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0810 03:31:02.516947 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:31:02.522330 (+ 5383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:31:02.522337 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:31:02.522366 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:31:02.522662 (+ 296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:31:02.522673 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:31:02.525308 (+ 2635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:31:02.525315 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:31:02.527595 (+ 2280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:31:02.527617 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:31:02.529074 (+ 1457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:31:02.529081 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:31:02.529262 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:31:02.529301 (+ 39us) client_negotiation.cc:769] Sending connection context
0810 03:31:02.529388 (+ 87us) client_negotiation.cc:240] Negotiation successful
0810 03:31:02.529559 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6486,"thread_start_us":6894,"threads_started":1}
I20260810 03:31:02.534780 24615 tablet_service.cc:1467] Processing CreateTablet for tablet 663072df88a04261b12dd903bc7e1fbe (DEFAULT_TABLE table=test-table [id=54cd887aaf734023a332d33c3bfeebeb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:31:02.536733 24615 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 663072df88a04261b12dd903bc7e1fbe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:31:02.541623 24733 tablet_bootstrap.cc:492] T 663072df88a04261b12dd903bc7e1fbe P d896337cd18049b094acb8c47b3f6e18: Bootstrap starting.
I20260810 03:31:02.542769 24735 tablet_bootstrap.cc:492] T 663072df88a04261b12dd903bc7e1fbe P a6279bfc96804d3798e8aa48b17a84d5: Bootstrap starting.
I20260810 03:31:02.546887 24735 tablet_bootstrap.cc:654] T 663072df88a04261b12dd903bc7e1fbe P a6279bfc96804d3798e8aa48b17a84d5: Neither blocks nor log segments found. Creating new log.
I20260810 03:31:02.547005 24733 tablet_bootstrap.cc:654] T 663072df88a04261b12dd903bc7e1fbe P d896337cd18049b094acb8c47b3f6e18: Neither blocks nor log segments found. Creating new log.
I20260810 03:31:02.548482 24733 log.cc:824] T 663072df88a04261b12dd903bc7e1fbe P d896337cd18049b094acb8c47b3f6e18: Log is configured to *not* fsync() on all Append() calls
I20260810 03:31:02.548607 24735 log.cc:824] T 663072df88a04261b12dd903bc7e1fbe P a6279bfc96804d3798e8aa48b17a84d5: Log is configured to *not* fsync() on all Append() calls
I20260810 03:31:02.552417 24733 tablet_bootstrap.cc:492] T 663072df88a04261b12dd903bc7e1fbe P d896337cd18049b094acb8c47b3f6e18: No bootstrap required, opened a new log
I20260810 03:31:02.552755 24735 tablet_bootstrap.cc:492] T 663072df88a04261b12dd903bc7e1fbe P a6279bfc96804d3798e8aa48b17a84d5: No bootstrap required, opened a new log
I20260810 03:31:02.552750 24733 ts_tablet_manager.cc:1397] T 663072df88a04261b12dd903bc7e1fbe P d896337cd18049b094acb8c47b3f6e18: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20260810 03:31:02.553030 24735 ts_tablet_manager.cc:1397] T 663072df88a04261b12dd903bc7e1fbe P a6279bfc96804d3798e8aa48b17a84d5: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.005s
I20260810 03:31:02.556849 24738 tablet_bootstrap.cc:492] T 663072df88a04261b12dd903bc7e1fbe P 506789d474ef48f1b4ec1d0839a3ca6c: Bootstrap starting.
I20260810 03:31:02.561008 24738 tablet_bootstrap.cc:654] T 663072df88a04261b12dd903bc7e1fbe P 506789d474ef48f1b4ec1d0839a3ca6c: Neither blocks nor log segments found. Creating new log.
I20260810 03:31:02.562582 24738 log.cc:824] T 663072df88a04261b12dd903bc7e1fbe P 506789d474ef48f1b4ec1d0839a3ca6c: Log is configured to *not* fsync() on all Append() calls
I20260810 03:31:02.562537 24733 raft_consensus.cc:348] T 663072df88a04261b12dd903bc7e1fbe P d896337cd18049b094acb8c47b3f6e18 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6279bfc96804d3798e8aa48b17a84d5" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 37673 } } peers { permanent_uuid: "d896337cd18049b094acb8c47b3f6e18" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 42953 } } peers { permanent_uuid: "506789d474ef48f1b4ec1d0839a3ca6c" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 42921 } }
I20260810 03:31:02.562567 24735 raft_consensus.cc:348] T 663072df88a04261b12dd903bc7e1fbe P a6279bfc96804d3798e8aa48b17a84d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6279bfc96804d3798e8aa48b17a84d5" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 37673 } } peers { permanent_uuid: "d896337cd18049b094acb8c47b3f6e18" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 42953 } } peers { permanent_uuid: "506789d474ef48f1b4ec1d0839a3ca6c" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 42921 } }
I20260810 03:31:02.562997 24733 raft_consensus.cc:374] T 663072df88a04261b12dd903bc7e1fbe P d896337cd18049b094acb8c47b3f6e18 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:31:02.563009 24735 raft_consensus.cc:374] T 663072df88a04261b12dd903bc7e1fbe P a6279bfc96804d3798e8aa48b17a84d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:31:02.563107 24735 raft_consensus.cc:729] T 663072df88a04261b12dd903bc7e1fbe P a6279bfc96804d3798e8aa48b17a84d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6279bfc96804d3798e8aa48b17a84d5, State: Initialized, Role: FOLLOWER
I20260810 03:31:02.563105 24733 raft_consensus.cc:729] T 663072df88a04261b12dd903bc7e1fbe P d896337cd18049b094acb8c47b3f6e18 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d896337cd18049b094acb8c47b3f6e18, State: Initialized, Role: FOLLOWER
I20260810 03:31:02.563848 24733 consensus_queue.cc:260] T 663072df88a04261b12dd903bc7e1fbe P d896337cd18049b094acb8c47b3f6e18 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6279bfc96804d3798e8aa48b17a84d5" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 37673 } } peers { permanent_uuid: "d896337cd18049b094acb8c47b3f6e18" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 42953 } } peers { permanent_uuid: "506789d474ef48f1b4ec1d0839a3ca6c" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 42921 } }
I20260810 03:31:02.563910 24735 consensus_queue.cc:260] T 663072df88a04261b12dd903bc7e1fbe P a6279bfc96804d3798e8aa48b17a84d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6279bfc96804d3798e8aa48b17a84d5" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 37673 } } peers { permanent_uuid: "d896337cd18049b094acb8c47b3f6e18" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 42953 } } peers { permanent_uuid: "506789d474ef48f1b4ec1d0839a3ca6c" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 42921 } }
I20260810 03:31:02.564934 24503 heartbeater.cc:503] Master 127.27.161.62:45507 was elected leader, sending a full tablet report...
I20260810 03:31:02.565670 24733 ts_tablet_manager.cc:1428] T 663072df88a04261b12dd903bc7e1fbe P d896337cd18049b094acb8c47b3f6e18: Time spent starting tablet: real 0.013s user 0.008s sys 0.005s
I20260810 03:31:02.567601 24735 ts_tablet_manager.cc:1428] T 663072df88a04261b12dd903bc7e1fbe P a6279bfc96804d3798e8aa48b17a84d5: Time spent starting tablet: real 0.014s user 0.014s sys 0.002s
I20260810 03:31:02.568866 24738 tablet_bootstrap.cc:492] T 663072df88a04261b12dd903bc7e1fbe P 506789d474ef48f1b4ec1d0839a3ca6c: No bootstrap required, opened a new log
I20260810 03:31:02.569198 24738 ts_tablet_manager.cc:1397] T 663072df88a04261b12dd903bc7e1fbe P 506789d474ef48f1b4ec1d0839a3ca6c: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.003s
I20260810 03:31:02.577287 24738 raft_consensus.cc:348] T 663072df88a04261b12dd903bc7e1fbe P 506789d474ef48f1b4ec1d0839a3ca6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6279bfc96804d3798e8aa48b17a84d5" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 37673 } } peers { permanent_uuid: "d896337cd18049b094acb8c47b3f6e18" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 42953 } } peers { permanent_uuid: "506789d474ef48f1b4ec1d0839a3ca6c" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 42921 } }
I20260810 03:31:02.577677 24738 raft_consensus.cc:374] T 663072df88a04261b12dd903bc7e1fbe P 506789d474ef48f1b4ec1d0839a3ca6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:31:02.577776 24738 raft_consensus.cc:729] T 663072df88a04261b12dd903bc7e1fbe P 506789d474ef48f1b4ec1d0839a3ca6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 506789d474ef48f1b4ec1d0839a3ca6c, State: Initialized, Role: FOLLOWER
I20260810 03:31:02.578459 24738 consensus_queue.cc:260] T 663072df88a04261b12dd903bc7e1fbe P 506789d474ef48f1b4ec1d0839a3ca6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6279bfc96804d3798e8aa48b17a84d5" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 37673 } } peers { permanent_uuid: "d896337cd18049b094acb8c47b3f6e18" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 42953 } } peers { permanent_uuid: "506789d474ef48f1b4ec1d0839a3ca6c" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 42921 } }
I20260810 03:31:02.579536 24707 heartbeater.cc:503] Master 127.27.161.62:45507 was elected leader, sending a full tablet report...
I20260810 03:31:02.580018 24738 ts_tablet_manager.cc:1428] T 663072df88a04261b12dd903bc7e1fbe P 506789d474ef48f1b4ec1d0839a3ca6c: Time spent starting tablet: real 0.011s user 0.012s sys 0.000s
W20260810 03:31:02.597134 24504 tablet.cc:2369] T 663072df88a04261b12dd903bc7e1fbe P d896337cd18049b094acb8c47b3f6e18: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 03:31:02.671653 24708 tablet.cc:2369] T 663072df88a04261b12dd903bc7e1fbe P 506789d474ef48f1b4ec1d0839a3ca6c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 03:31:02.705426 24741 raft_consensus.cc:482] T 663072df88a04261b12dd903bc7e1fbe P a6279bfc96804d3798e8aa48b17a84d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 03:31:02.705741 24741 raft_consensus.cc:504] T 663072df88a04261b12dd903bc7e1fbe P a6279bfc96804d3798e8aa48b17a84d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6279bfc96804d3798e8aa48b17a84d5" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 37673 } } peers { permanent_uuid: "d896337cd18049b094acb8c47b3f6e18" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 42953 } } peers { permanent_uuid: "506789d474ef48f1b4ec1d0839a3ca6c" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 42921 } }
I20260810 03:31:02.707566 24741 leader_election.cc:288] T 663072df88a04261b12dd903bc7e1fbe P a6279bfc96804d3798e8aa48b17a84d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d896337cd18049b094acb8c47b3f6e18 (127.27.161.2:42953), 506789d474ef48f1b4ec1d0839a3ca6c (127.27.161.3:42921)
I20260810 03:31:02.716447 24731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:31:02.708148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.161.1:35813 (local address 127.27.161.3:42921)
0810 03:31:02.708598 (+ 450us) server_negotiation.cc:206] Beginning negotiation
0810 03:31:02.708604 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:31:02.708995 (+ 391us) server_negotiation.cc:407] Connection header received
0810 03:31:02.709158 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:31:02.709163 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:31:02.709236 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:31:02.709320 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:31:02.710053 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:31:02.711183 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:31:02.714059 (+ 2876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:31:02.715316 (+ 1257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:31:02.715490 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:31:02.715572 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0810 03:31:02.716025 (+ 453us) server_negotiation.cc:299] Negotiation successful
0810 03:31:02.716170 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":305}
I20260810 03:31:02.716671 24747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:31:02.708016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.3:42921 (local address 127.27.161.1:35813)
0810 03:31:02.708906 (+ 890us) negotiation.cc:107] Waiting for socket to connect
0810 03:31:02.708921 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 03:31:02.709021 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:31:02.709332 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:31:02.709337 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:31:02.709365 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:31:02.709851 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:31:02.709868 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:31:02.711299 (+ 1431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:31:02.711304 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:31:02.713815 (+ 2511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:31:02.713831 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:31:02.715613 (+ 1782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:31:02.715618 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:31:02.715818 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:31:02.715875 (+ 57us) client_negotiation.cc:769] Sending connection context
0810 03:31:02.715976 (+ 101us) client_negotiation.cc:240] Negotiation successful
0810 03:31:02.716167 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":782,"mutex_wait_us":108,"thread_start_us":618,"threads_started":1}
I20260810 03:31:02.716449 24727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:31:02.708064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.161.1:43365 (local address 127.27.161.2:42953)
0810 03:31:02.708418 (+ 354us) server_negotiation.cc:206] Beginning negotiation
0810 03:31:02.708426 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 03:31:02.708835 (+ 409us) server_negotiation.cc:407] Connection header received
0810 03:31:02.709087 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:31:02.709106 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:31:02.709193 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:31:02.709303 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:31:02.710061 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:31:02.711175 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:31:02.714045 (+ 2870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:31:02.715322 (+ 1277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:31:02.715496 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:31:02.715572 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0810 03:31:02.716026 (+ 454us) server_negotiation.cc:299] Negotiation successful
0810 03:31:02.716170 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169}
I20260810 03:31:02.716697 24746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:31:02.707831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.2:42953 (local address 127.27.161.1:43365)
0810 03:31:02.708695 (+ 864us) negotiation.cc:107] Waiting for socket to connect
0810 03:31:02.708719 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0810 03:31:02.708906 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:31:02.709332 (+ 426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:31:02.709337 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:31:02.709434 (+ 97us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:31:02.709929 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:31:02.709939 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:31:02.711302 (+ 1363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:31:02.711306 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:31:02.713817 (+ 2511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:31:02.713831 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:31:02.715598 (+ 1767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:31:02.715606 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:31:02.715848 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:31:02.715894 (+ 46us) client_negotiation.cc:769] Sending connection context
0810 03:31:02.715976 (+ 82us) client_negotiation.cc:240] Negotiation successful
0810 03:31:02.716167 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":686,"mutex_wait_us":297,"spinlock_wait_cycles":24832,"thread_start_us":471,"threads_started":1}
I20260810 03:31:02.718012 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "663072df88a04261b12dd903bc7e1fbe" candidate_uuid: "a6279bfc96804d3798e8aa48b17a84d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d896337cd18049b094acb8c47b3f6e18" is_pre_election: true
I20260810 03:31:02.718017 24643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "663072df88a04261b12dd903bc7e1fbe" candidate_uuid: "a6279bfc96804d3798e8aa48b17a84d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "506789d474ef48f1b4ec1d0839a3ca6c" is_pre_election: true
I20260810 03:31:02.718631 24430 raft_consensus.cc:2454] T 663072df88a04261b12dd903bc7e1fbe P d896337cd18049b094acb8c47b3f6e18 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a6279bfc96804d3798e8aa48b17a84d5 in term 0.
I20260810 03:31:02.718636 24643 raft_consensus.cc:2454] T 663072df88a04261b12dd903bc7e1fbe P 506789d474ef48f1b4ec1d0839a3ca6c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a6279bfc96804d3798e8aa48b17a84d5 in term 0.
I20260810 03:31:02.719457 24113 leader_election.cc:302] T 663072df88a04261b12dd903bc7e1fbe P a6279bfc96804d3798e8aa48b17a84d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 506789d474ef48f1b4ec1d0839a3ca6c, a6279bfc96804d3798e8aa48b17a84d5, d896337cd18049b094acb8c47b3f6e18; no voters:
I20260810 03:31:02.719906 24741 raft_consensus.cc:2781] T 663072df88a04261b12dd903bc7e1fbe P a6279bfc96804d3798e8aa48b17a84d5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 03:31:02.720075 24741 raft_consensus.cc:482] T 663072df88a04261b12dd903bc7e1fbe P a6279bfc96804d3798e8aa48b17a84d5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 03:31:02.720182 24741 raft_consensus.cc:3037] T 663072df88a04261b12dd903bc7e1fbe P a6279bfc96804d3798e8aa48b17a84d5 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:31:02.722549 24741 raft_consensus.cc:504] T 663072df88a04261b12dd903bc7e1fbe P a6279bfc96804d3798e8aa48b17a84d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6279bfc96804d3798e8aa48b17a84d5" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 37673 } } peers { permanent_uuid: "d896337cd18049b094acb8c47b3f6e18" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 42953 } } peers { permanent_uuid: "506789d474ef48f1b4ec1d0839a3ca6c" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 42921 } }
I20260810 03:31:02.723244 24741 leader_election.cc:288] T 663072df88a04261b12dd903bc7e1fbe P a6279bfc96804d3798e8aa48b17a84d5 [CANDIDATE]: Term 1 election: Requested vote from peers d896337cd18049b094acb8c47b3f6e18 (127.27.161.2:42953), 506789d474ef48f1b4ec1d0839a3ca6c (127.27.161.3:42921)
I20260810 03:31:02.723801 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "663072df88a04261b12dd903bc7e1fbe" candidate_uuid: "a6279bfc96804d3798e8aa48b17a84d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d896337cd18049b094acb8c47b3f6e18"
I20260810 03:31:02.723846 24643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "663072df88a04261b12dd903bc7e1fbe" candidate_uuid: "a6279bfc96804d3798e8aa48b17a84d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "506789d474ef48f1b4ec1d0839a3ca6c"
I20260810 03:31:02.723994 24430 raft_consensus.cc:3037] T 663072df88a04261b12dd903bc7e1fbe P d896337cd18049b094acb8c47b3f6e18 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:31:02.724074 24643 raft_consensus.cc:3037] T 663072df88a04261b12dd903bc7e1fbe P 506789d474ef48f1b4ec1d0839a3ca6c [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:31:02.726331 24430 raft_consensus.cc:2454] T 663072df88a04261b12dd903bc7e1fbe P d896337cd18049b094acb8c47b3f6e18 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6279bfc96804d3798e8aa48b17a84d5 in term 1.
I20260810 03:31:02.726671 24643 raft_consensus.cc:2454] T 663072df88a04261b12dd903bc7e1fbe P 506789d474ef48f1b4ec1d0839a3ca6c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6279bfc96804d3798e8aa48b17a84d5 in term 1.
I20260810 03:31:02.726840 24114 leader_election.cc:302] T 663072df88a04261b12dd903bc7e1fbe P a6279bfc96804d3798e8aa48b17a84d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a6279bfc96804d3798e8aa48b17a84d5, d896337cd18049b094acb8c47b3f6e18; no voters:
I20260810 03:31:02.727227 24741 raft_consensus.cc:2781] T 663072df88a04261b12dd903bc7e1fbe P a6279bfc96804d3798e8aa48b17a84d5 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:31:02.727725 24741 raft_consensus.cc:686] T 663072df88a04261b12dd903bc7e1fbe P a6279bfc96804d3798e8aa48b17a84d5 [term 1 LEADER]: Becoming Leader. State: Replica: a6279bfc96804d3798e8aa48b17a84d5, State: Running, Role: LEADER
I20260810 03:31:02.728227 24741 consensus_queue.cc:237] T 663072df88a04261b12dd903bc7e1fbe P a6279bfc96804d3798e8aa48b17a84d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6279bfc96804d3798e8aa48b17a84d5" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 37673 } } peers { permanent_uuid: "d896337cd18049b094acb8c47b3f6e18" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 42953 } } peers { permanent_uuid: "506789d474ef48f1b4ec1d0839a3ca6c" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 42921 } }
I20260810 03:31:02.733086 23997 catalog_manager.cc:5495] T 663072df88a04261b12dd903bc7e1fbe P a6279bfc96804d3798e8aa48b17a84d5 reported cstate change: term changed from 0 to 1, leader changed from <none> to a6279bfc96804d3798e8aa48b17a84d5 (127.27.161.1). New cstate: current_term: 1 leader_uuid: "a6279bfc96804d3798e8aa48b17a84d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6279bfc96804d3798e8aa48b17a84d5" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 37673 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d896337cd18049b094acb8c47b3f6e18" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 42953 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "506789d474ef48f1b4ec1d0839a3ca6c" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 42921 } health_report { overall_health: UNKNOWN } } }
W20260810 03:31:02.770385 24292 tablet.cc:2369] T 663072df88a04261b12dd903bc7e1fbe P a6279bfc96804d3798e8aa48b17a84d5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 03:31:02.771554 24729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:31:02.766037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33946 (local address 127.27.161.1:37673)
0810 03:31:02.766272 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0810 03:31:02.766278 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:31:02.766301 (+ 23us) server_negotiation.cc:407] Connection header received
0810 03:31:02.766574 (+ 273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:31:02.766579 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:31:02.766701 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:31:02.766843 (+ 142us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 03:31:02.767215 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:31:02.767946 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:31:02.769305 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:31:02.769709 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:31:02.769798 (+ 89us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:31:02.770330 (+ 532us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 03:31:02.770921 (+ 591us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 03:31:02.771066 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0810 03:31:02.771241 (+ 175us) server_negotiation.cc:299] Negotiation successful
0810 03:31:02.771284 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260810 03:31:02.780095 24643 raft_consensus.cc:1261] T 663072df88a04261b12dd903bc7e1fbe P 506789d474ef48f1b4ec1d0839a3ca6c [term 1 FOLLOWER]: Refusing update from remote peer a6279bfc96804d3798e8aa48b17a84d5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:31:02.780122 24430 raft_consensus.cc:1261] T 663072df88a04261b12dd903bc7e1fbe P d896337cd18049b094acb8c47b3f6e18 [term 1 FOLLOWER]: Refusing update from remote peer a6279bfc96804d3798e8aa48b17a84d5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:31:02.781149 24748 consensus_queue.cc:1035] T 663072df88a04261b12dd903bc7e1fbe P a6279bfc96804d3798e8aa48b17a84d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d896337cd18049b094acb8c47b3f6e18" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 42953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:31:02.781508 24741 consensus_queue.cc:1035] T 663072df88a04261b12dd903bc7e1fbe P a6279bfc96804d3798e8aa48b17a84d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "506789d474ef48f1b4ec1d0839a3ca6c" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 42921 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:31:02.793942 24752 mvcc.cc:205] Tried to move back new op lower bound from 7316818586736455680 to 7316818586538938368. Current Snapshot: MvccSnapshot[applied={T|T < 7316818586736455680}]
I20260810 03:31:02.813145 23997 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33652:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 03:31:02.813779 23997 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33652:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 03:31:02.819204 23997 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0ec4d8ba88a94fd8838397056fac4bc4: Sending DeleteTablet for 3 replicas of tablet 663072df88a04261b12dd903bc7e1fbe
I20260810 03:31:02.820910 24615 tablet_service.cc:1514] Processing DeleteTablet for tablet 663072df88a04261b12dd903bc7e1fbe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:31:02 UTC) from {username='kudu', principal='kudu/127.27.161.62@KRBTEST.COM'} at 127.0.0.1:33652
I20260810 03:31:02.820870 24403 tablet_service.cc:1514] Processing DeleteTablet for tablet 663072df88a04261b12dd903bc7e1fbe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:31:02 UTC) from {username='kudu', principal='kudu/127.27.161.62@KRBTEST.COM'} at 127.0.0.1:44024
I20260810 03:31:02.821197 24188 tablet_service.cc:1514] Processing DeleteTablet for tablet 663072df88a04261b12dd903bc7e1fbe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:31:02 UTC) from {username='kudu', principal='kudu/127.27.161.62@KRBTEST.COM'} at 127.0.0.1:33942
I20260810 03:31:02.822552 24773 tablet_replica.cc:330] T 663072df88a04261b12dd903bc7e1fbe P d896337cd18049b094acb8c47b3f6e18: stopping tablet replica
I20260810 03:31:02.822811 24771 tablet_replica.cc:330] T 663072df88a04261b12dd903bc7e1fbe P 506789d474ef48f1b4ec1d0839a3ca6c: stopping tablet replica
I20260810 03:31:02.824229 24771 raft_consensus.cc:2229] T 663072df88a04261b12dd903bc7e1fbe P 506789d474ef48f1b4ec1d0839a3ca6c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 03:31:02.824965 24771 raft_consensus.cc:2258] T 663072df88a04261b12dd903bc7e1fbe P 506789d474ef48f1b4ec1d0839a3ca6c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:31:02.826414 24773 raft_consensus.cc:2229] T 663072df88a04261b12dd903bc7e1fbe P d896337cd18049b094acb8c47b3f6e18 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 03:31:02.826979 28292 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iBH5/build/asan/bin/kudu pid 24087
I20260810 03:31:02.827139 24773 raft_consensus.cc:2258] T 663072df88a04261b12dd903bc7e1fbe P d896337cd18049b094acb8c47b3f6e18 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:31:02.828168 24772 tablet_replica.cc:330] T 663072df88a04261b12dd903bc7e1fbe P a6279bfc96804d3798e8aa48b17a84d5: stopping tablet replica
I20260810 03:31:02.828658 24772 raft_consensus.cc:2229] T 663072df88a04261b12dd903bc7e1fbe P a6279bfc96804d3798e8aa48b17a84d5 [term 1 LEADER]: Raft consensus shutting down.
I20260810 03:31:02.829424 24772 raft_consensus.cc:2258] T 663072df88a04261b12dd903bc7e1fbe P a6279bfc96804d3798e8aa48b17a84d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:31:02.830462 24773 ts_tablet_manager.cc:1905] T 663072df88a04261b12dd903bc7e1fbe P d896337cd18049b094acb8c47b3f6e18: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:31:02.831626 24771 ts_tablet_manager.cc:1905] T 663072df88a04261b12dd903bc7e1fbe P 506789d474ef48f1b4ec1d0839a3ca6c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:31:02.831676 24772 ts_tablet_manager.cc:1905] T 663072df88a04261b12dd903bc7e1fbe P a6279bfc96804d3798e8aa48b17a84d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:31:02.839533 24772 ts_tablet_manager.cc:1918] T 663072df88a04261b12dd903bc7e1fbe P a6279bfc96804d3798e8aa48b17a84d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 03:31:02.839761 24772 log.cc:1196] T 663072df88a04261b12dd903bc7e1fbe P a6279bfc96804d3798e8aa48b17a84d5: Deleting WAL directory at /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786332336319544-28292-0/minicluster-data/ts-0/wal/wals/663072df88a04261b12dd903bc7e1fbe
I20260810 03:31:02.840391 24772 ts_tablet_manager.cc:1939] T 663072df88a04261b12dd903bc7e1fbe P a6279bfc96804d3798e8aa48b17a84d5: Deleting consensus metadata
I20260810 03:31:02.842475 23972 catalog_manager.cc:4841] TS a6279bfc96804d3798e8aa48b17a84d5 (127.27.161.1:37673): tablet 663072df88a04261b12dd903bc7e1fbe (table test-table [id=54cd887aaf734023a332d33c3bfeebeb]) successfully deleted
I20260810 03:31:02.844197 24773 ts_tablet_manager.cc:1918] T 663072df88a04261b12dd903bc7e1fbe P d896337cd18049b094acb8c47b3f6e18: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 03:31:02.844413 24773 log.cc:1196] T 663072df88a04261b12dd903bc7e1fbe P d896337cd18049b094acb8c47b3f6e18: Deleting WAL directory at /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786332336319544-28292-0/minicluster-data/ts-1/wal/wals/663072df88a04261b12dd903bc7e1fbe
I20260810 03:31:02.844995 24773 ts_tablet_manager.cc:1939] T 663072df88a04261b12dd903bc7e1fbe P d896337cd18049b094acb8c47b3f6e18: Deleting consensus metadata
I20260810 03:31:02.846923 24771 ts_tablet_manager.cc:1918] T 663072df88a04261b12dd903bc7e1fbe P 506789d474ef48f1b4ec1d0839a3ca6c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 03:31:02.847182 24771 log.cc:1196] T 663072df88a04261b12dd903bc7e1fbe P 506789d474ef48f1b4ec1d0839a3ca6c: Deleting WAL directory at /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786332336319544-28292-0/minicluster-data/ts-2/wal/wals/663072df88a04261b12dd903bc7e1fbe
I20260810 03:31:02.847301 23974 catalog_manager.cc:4841] TS d896337cd18049b094acb8c47b3f6e18 (127.27.161.2:42953): tablet 663072df88a04261b12dd903bc7e1fbe (table test-table [id=54cd887aaf734023a332d33c3bfeebeb]) successfully deleted
I20260810 03:31:02.847887 24771 ts_tablet_manager.cc:1939] T 663072df88a04261b12dd903bc7e1fbe P 506789d474ef48f1b4ec1d0839a3ca6c: Deleting consensus metadata
I20260810 03:31:02.849915 23972 catalog_manager.cc:4841] TS 506789d474ef48f1b4ec1d0839a3ca6c (127.27.161.3:42921): tablet 663072df88a04261b12dd903bc7e1fbe (table test-table [id=54cd887aaf734023a332d33c3bfeebeb]) successfully deleted
Aug 10 03:31:02 dist-test-slave-vpvm krb5kdc[23928](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332659, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.161.1@KRBTEST.COM
I20260810 03:31:02.862033 24729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:31:02.828173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33954 (local address 127.27.161.1:37673)
0810 03:31:02.828369 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0810 03:31:02.828376 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:31:02.840067 (+ 11691us) server_negotiation.cc:407] Connection header received
0810 03:31:02.840643 (+ 576us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:31:02.840649 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:31:02.840752 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:31:02.840834 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:31:02.847844 (+ 7010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:31:02.849105 (+ 1261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:31:02.851023 (+ 1918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:31:02.851488 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:31:02.851617 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:31:02.855880 (+ 4263us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:31:02.855898 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:31:02.855915 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:31:02.855956 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:31:02.858949 (+ 2993us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:31:02.859709 (+ 760us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:31:02.859716 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:31:02.859719 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:31:02.859874 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:31:02.860255 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:31:02.860258 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:31:02.860260 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:31:02.860637 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:31:02.860926 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0810 03:31:02.861294 (+ 368us) server_negotiation.cc:299] Negotiation successful
0810 03:31:02.861557 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260810 03:31:02.862730 24282 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:31:02.966754 28292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iBH5/build/asan/bin/kudu with pid 24087
I20260810 03:31:02.998807 28292 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iBH5/build/asan/bin/kudu pid 24299
Aug 10 03:31:03 dist-test-slave-vpvm krb5kdc[23928](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332659, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.161.2@KRBTEST.COM
I20260810 03:31:03.014004 24727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:31:02.999663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44038 (local address 127.27.161.2:42953)
0810 03:31:02.999850 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0810 03:31:02.999855 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:31:02.999875 (+ 20us) server_negotiation.cc:407] Connection header received
0810 03:31:02.999961 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:31:02.999965 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:31:03.000043 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:31:03.000126 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:31:03.001490 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:31:03.002328 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:31:03.003641 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:31:03.004030 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:31:03.004119 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:31:03.008461 (+ 4342us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:31:03.008476 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:31:03.008501 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:31:03.008547 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:31:03.011304 (+ 2757us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:31:03.012006 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:31:03.012015 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:31:03.012018 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:31:03.012158 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:31:03.012520 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:31:03.012524 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:31:03.012526 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:31:03.012970 (+ 444us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:31:03.013206 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0810 03:31:03.013466 (+ 260us) server_negotiation.cc:299] Negotiation successful
0810 03:31:03.013648 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260810 03:31:03.014794 24497 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:31:03.105316 28292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iBH5/build/asan/bin/kudu with pid 24299
I20260810 03:31:03.126709 28292 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iBH5/build/asan/bin/kudu pid 24513
Aug 10 03:31:03 dist-test-slave-vpvm krb5kdc[23928](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332659, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.161.3@KRBTEST.COM
I20260810 03:31:03.142347 24731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:31:03.127530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33668 (local address 127.27.161.3:42921)
0810 03:31:03.127731 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0810 03:31:03.127738 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:31:03.127763 (+ 25us) server_negotiation.cc:407] Connection header received
0810 03:31:03.127969 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:31:03.127975 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:31:03.128059 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:31:03.128158 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:31:03.129484 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:31:03.130300 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:31:03.131606 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:31:03.131992 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:31:03.132115 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:31:03.136572 (+ 4457us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:31:03.136590 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:31:03.136609 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:31:03.136650 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:31:03.139439 (+ 2789us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:31:03.140210 (+ 771us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:31:03.140218 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:31:03.140221 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:31:03.140376 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:31:03.140752 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:31:03.140756 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:31:03.140758 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:31:03.141142 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:31:03.141431 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0810 03:31:03.141753 (+ 322us) server_negotiation.cc:299] Negotiation successful
0810 03:31:03.141998 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260810 03:31:03.143024 24702 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:31:03.234130 28292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iBH5/build/asan/bin/kudu with pid 24513
I20260810 03:31:03.258657 28292 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iBH5/build/asan/bin/kudu pid 23950
I20260810 03:31:03.260170 24046 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:31:03.373278 28292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iBH5/build/asan/bin/kudu with pid 23950
2026-08-10T03:31:03Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (4864 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-08-10T03:31:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T03:31:03Z Disabled control of system clock
I20260810 03:31:03.481851 28292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iBH5/build/asan/bin/kudu
/tmp/dist-test-task24iBH5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786332336319544-28292-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786332336319544-28292-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786332336319544-28292-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786332336319544-28292-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.161.62:43637
--webserver_interface=127.27.161.62
--webserver_port=0
--builtin_ntp_servers=127.27.161.20:38599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.161.62:43637
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260810 03:31:03.841136 24786 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:31:03.841470 24786 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:31:03.841606 24786 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:31:03.849335 24786 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:31:03.849472 24786 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:31:03.849520 24786 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:31:03.849612 24786 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:31:03.849675 24786 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:31:03.859552 24786 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.161.20:38599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786332336319544-28292-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786332336319544-28292-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.161.62:43637
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.161.62:43637
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786332336319544-28292-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.161.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786332336319544-28292-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:31:03.860972 24786 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:31:03.862914 24786 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:31:03.875310 24793 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:31:03.875372 24794 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:31:03.875306 24797 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:31:03.876185 24786 server_base.cc:1034] running on GCE node
I20260810 03:31:03.877053 24786 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:31:03.878705 24786 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:31:03.879920 24786 hybrid_clock.cc:648] HybridClock initialized: now 1786332663879873 us; error 46 us; skew 500 ppm
I20260810 03:31:03.880303 24786 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 03:31:03.882884 24786 webserver.cc:460] Webserver started at http://127.27.161.62:39875/ using document root <none> and password file <none>
I20260810 03:31:03.883658 24786 fs_manager.cc:359] Metadata directory not provided
I20260810 03:31:03.883773 24786 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:31:03.884117 24786 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:31:03.887365 24786 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786332336319544-28292-0/minicluster-data/master-0/data/instance:
uuid: "306967a259714361bb8308b13373dc85"
format_stamp: "Formatted at 2026-08-10 03:31:03 on dist-test-slave-vpvm"
server_key: "dcd7350a5bd41e600ad71e434d41a9d7"
server_key_iv: "5abf63167a8ec1b2c98ddfe25813825e"
server_key_version: "encryptionkey@0"
I20260810 03:31:03.888278 24786 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786332336319544-28292-0/minicluster-data/master-0/wal/instance:
uuid: "306967a259714361bb8308b13373dc85"
format_stamp: "Formatted at 2026-08-10 03:31:03 on dist-test-slave-vpvm"
server_key: "dcd7350a5bd41e600ad71e434d41a9d7"
server_key_iv: "5abf63167a8ec1b2c98ddfe25813825e"
server_key_version: "encryptionkey@0"
I20260810 03:31:03.894170 24786 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260810 03:31:03.898070 24805 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:31:03.899461 24786 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 03:31:03.899650 24786 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786332336319544-28292-0/minicluster-data/master-0/data,/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786332336319544-28292-0/minicluster-data/master-0/wal
uuid: "306967a259714361bb8308b13373dc85"
format_stamp: "Formatted at 2026-08-10 03:31:03 on dist-test-slave-vpvm"
server_key: "dcd7350a5bd41e600ad71e434d41a9d7"
server_key_iv: "5abf63167a8ec1b2c98ddfe25813825e"
server_key_version: "encryptionkey@0"
I20260810 03:31:03.899865 24786 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786332336319544-28292-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786332336319544-28292-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786332336319544-28292-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:31:03.925661 24786 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:31:03.926635 24786 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:31:03.926903 24786 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:31:03.945246 24786 rpc_server.cc:307] RPC server started. Bound to: 127.27.161.62:43637
I20260810 03:31:03.945272 24889 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.161.62:43637 every 8 connection(s)
I20260810 03:31:03.946703 24786 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786332336319544-28292-0/minicluster-data/master-0/data/info.pb
I20260810 03:31:03.950577 24890 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:31:03.956543 28292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iBH5/build/asan/bin/kudu as pid 24786
I20260810 03:31:03.956707 28292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786332336319544-28292-0/minicluster-data/master-0/wal/instance
I20260810 03:31:03.957129 28292 external_mini_cluster.cc:1428] Setting key f6fd1f2071fe344a20fd3469676b83fd
I20260810 03:31:03.962527 24890 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 306967a259714361bb8308b13373dc85: Bootstrap starting.
I20260810 03:31:03.966897 24890 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 306967a259714361bb8308b13373dc85: Neither blocks nor log segments found. Creating new log.
I20260810 03:31:03.968019 24890 log.cc:824] T 00000000000000000000000000000000 P 306967a259714361bb8308b13373dc85: Log is configured to *not* fsync() on all Append() calls
I20260810 03:31:03.971312 24890 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 306967a259714361bb8308b13373dc85: No bootstrap required, opened a new log
I20260810 03:31:03.973157 24894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:31:03.959240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56762 (local address 127.27.161.62:43637)
0810 03:31:03.960427 (+ 1187us) server_negotiation.cc:206] Beginning negotiation
0810 03:31:03.960442 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 03:31:03.960497 (+ 55us) server_negotiation.cc:407] Connection header received
0810 03:31:03.962956 (+ 2459us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:31:03.962990 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:31:03.963652 (+ 662us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:31:03.964137 (+ 485us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:31:03.964755 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:31:03.966723 (+ 1968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:31:03.968306 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:31:03.969350 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:31:03.969564 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:31:03.970736 (+ 1172us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:31:03.970748 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:31:03.970765 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:31:03.971319 (+ 554us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:31:03.971408 (+ 89us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:31:03.971440 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:31:03.971554 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0810 03:31:03.971791 (+ 237us) server_negotiation.cc:299] Negotiation successful
0810 03:31:03.972131 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":600,"thread_start_us":358,"threads_started":1}
I20260810 03:31:03.980049 24890 raft_consensus.cc:348] T 00000000000000000000000000000000 P 306967a259714361bb8308b13373dc85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "306967a259714361bb8308b13373dc85" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 43637 } }
I20260810 03:31:03.980453 24890 raft_consensus.cc:374] T 00000000000000000000000000000000 P 306967a259714361bb8308b13373dc85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:31:03.980549 24890 raft_consensus.cc:729] T 00000000000000000000000000000000 P 306967a259714361bb8308b13373dc85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 306967a259714361bb8308b13373dc85, State: Initialized, Role: FOLLOWER
I20260810 03:31:03.981117 24890 consensus_queue.cc:260] T 00000000000000000000000000000000 P 306967a259714361bb8308b13373dc85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "306967a259714361bb8308b13373dc85" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 43637 } }
I20260810 03:31:03.981346 24890 raft_consensus.cc:388] T 00000000000000000000000000000000 P 306967a259714361bb8308b13373dc85 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 03:31:03.981469 24890 raft_consensus.cc:482] T 00000000000000000000000000000000 P 306967a259714361bb8308b13373dc85 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 03:31:03.981624 24890 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 306967a259714361bb8308b13373dc85 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:31:03.983942 24890 raft_consensus.cc:504] T 00000000000000000000000000000000 P 306967a259714361bb8308b13373dc85 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "306967a259714361bb8308b13373dc85" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 43637 } }
I20260810 03:31:03.984596 24890 leader_election.cc:302] T 00000000000000000000000000000000 P 306967a259714361bb8308b13373dc85 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 306967a259714361bb8308b13373dc85; no voters:
I20260810 03:31:03.985196 24890 leader_election.cc:288] T 00000000000000000000000000000000 P 306967a259714361bb8308b13373dc85 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 03:31:03.985390 24896 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 306967a259714361bb8308b13373dc85 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:31:03.986148 24896 raft_consensus.cc:686] T 00000000000000000000000000000000 P 306967a259714361bb8308b13373dc85 [term 1 LEADER]: Becoming Leader. State: Replica: 306967a259714361bb8308b13373dc85, State: Running, Role: LEADER
I20260810 03:31:03.986737 24896 consensus_queue.cc:237] T 00000000000000000000000000000000 P 306967a259714361bb8308b13373dc85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "306967a259714361bb8308b13373dc85" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 43637 } }
I20260810 03:31:03.987313 24890 sys_catalog.cc:564] T 00000000000000000000000000000000 P 306967a259714361bb8308b13373dc85 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:31:03.989697 24898 sys_catalog.cc:455] T 00000000000000000000000000000000 P 306967a259714361bb8308b13373dc85 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 306967a259714361bb8308b13373dc85. Latest consensus state: current_term: 1 leader_uuid: "306967a259714361bb8308b13373dc85" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "306967a259714361bb8308b13373dc85" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 43637 } } }
I20260810 03:31:03.990278 24898 sys_catalog.cc:458] T 00000000000000000000000000000000 P 306967a259714361bb8308b13373dc85 [sys.catalog]: This master's current role is: LEADER
I20260810 03:31:03.991218 24897 sys_catalog.cc:455] T 00000000000000000000000000000000 P 306967a259714361bb8308b13373dc85 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "306967a259714361bb8308b13373dc85" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "306967a259714361bb8308b13373dc85" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 43637 } } }
I20260810 03:31:03.992100 24897 sys_catalog.cc:458] T 00000000000000000000000000000000 P 306967a259714361bb8308b13373dc85 [sys.catalog]: This master's current role is: LEADER
I20260810 03:31:03.993088 24911 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 03:31:03.998806 24911 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 03:31:04.008595 24911 catalog_manager.cc:1347] Generated new cluster ID: 7f4205334890412cbd10a6a79a6aadd3
I20260810 03:31:04.008728 24911 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 03:31:04.033711 24911 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 03:31:04.035064 24911 catalog_manager.cc:1504] Loading token signing keys...
I20260810 03:31:04.040954 24911 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 306967a259714361bb8308b13373dc85: Generated new TSK 0
I20260810 03:31:04.041775 24911 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 03:31:04.057260 28292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iBH5/build/asan/bin/kudu
/tmp/dist-test-task24iBH5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786332336319544-28292-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786332336319544-28292-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786332336319544-28292-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786332336319544-28292-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.161.1:0
--local_ip_for_outbound_sockets=127.27.161.1
--webserver_interface=127.27.161.1
--webserver_port=0
--tserver_master_addrs=127.27.161.62:43637
--builtin_ntp_servers=127.27.161.20:38599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260810 03:31:04.447567 24926 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:31:04.447860 24926 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:31:04.447950 24926 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:31:04.455220 24926 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:31:04.455317 24926 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:31:04.455420 24926 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.161.1
I20260810 03:31:04.464582 24926 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.161.20:38599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786332336319544-28292-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786332336319544-28292-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.161.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786332336319544-28292-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.161.1
--webserver_port=0
--tserver_master_addrs=127.27.161.62:43637
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.161.1
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786332336319544-28292-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:31:04.465987 24926 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:31:04.467850 24926 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:31:04.481112 24937 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:31:04.481081 24940 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:31:04.481038 24938 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:31:04.482220 24926 server_base.cc:1034] running on GCE node
I20260810 03:31:04.482812 24926 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:31:04.483842 24926 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:31:04.485065 24926 hybrid_clock.cc:648] HybridClock initialized: now 1786332664485001 us; error 51 us; skew 500 ppm
I20260810 03:31:04.485405 24926 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 03:31:04.487879 24926 webserver.cc:460] Webserver started at http://127.27.161.1:44003/ using document root <none> and password file <none>
I20260810 03:31:04.488658 24926 fs_manager.cc:359] Metadata directory not provided
I20260810 03:31:04.488770 24926 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:31:04.489086 24926 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:31:04.492233 24926 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786332336319544-28292-0/minicluster-data/ts-0/data/instance:
uuid: "44e811ef403a4ff99f2056d86168de29"
format_stamp: "Formatted at 2026-08-10 03:31:04 on dist-test-slave-vpvm"
server_key: "f430fb55509879749f63702632750ed8"
server_key_iv: "4f1e4130b0bfeca4dbd76eab790625a1"
server_key_version: "encryptionkey@0"
I20260810 03:31:04.493052 24926 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786332336319544-28292-0/minicluster-data/ts-0/wal/instance:
uuid: "44e811ef403a4ff99f2056d86168de29"
format_stamp: "Formatted at 2026-08-10 03:31:04 on dist-test-slave-vpvm"
server_key: "f430fb55509879749f63702632750ed8"
server_key_iv: "4f1e4130b0bfeca4dbd76eab790625a1"
server_key_version: "encryptionkey@0"
I20260810 03:31:04.498600 24926 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260810 03:31:04.502542 24949 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:31:04.504244 24926 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260810 03:31:04.504458 24926 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786332336319544-28292-0/minicluster-data/ts-0/data,/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786332336319544-28292-0/minicluster-data/ts-0/wal
uuid: "44e811ef403a4ff99f2056d86168de29"
format_stamp: "Formatted at 2026-08-10 03:31:04 on dist-test-slave-vpvm"
server_key: "f430fb55509879749f63702632750ed8"
server_key_iv: "4f1e4130b0bfeca4dbd76eab790625a1"
server_key_version: "encryptionkey@0"
I20260810 03:31:04.504709 24926 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786332336319544-28292-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786332336319544-28292-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786332336319544-28292-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:31:04.533638 24926 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:31:04.534639 24926 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:31:04.534940 24926 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:31:04.536000 24926 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:31:04.537635 24926 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:31:04.537734 24926 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:31:04.537834 24926 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:31:04.537905 24926 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:31:04.571616 24926 rpc_server.cc:307] RPC server started. Bound to: 127.27.161.1:40327
I20260810 03:31:04.571646 25125 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.161.1:40327 every 8 connection(s)
I20260810 03:31:04.572997 24926 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786332336319544-28292-0/minicluster-data/ts-0/data/info.pb
I20260810 03:31:04.577916 28292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iBH5/build/asan/bin/kudu as pid 24926
I20260810 03:31:04.578125 28292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786332336319544-28292-0/minicluster-data/ts-0/wal/instance
I20260810 03:31:04.578723 28292 external_mini_cluster.cc:1428] Setting key de1ad17f7ab2535eb5495a0c185f24f2
I20260810 03:31:04.581566 28292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iBH5/build/asan/bin/kudu
/tmp/dist-test-task24iBH5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786332336319544-28292-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786332336319544-28292-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786332336319544-28292-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786332336319544-28292-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.161.2:0
--local_ip_for_outbound_sockets=127.27.161.2
--webserver_interface=127.27.161.2
--webserver_port=0
--tserver_master_addrs=127.27.161.62:43637
--builtin_ntp_servers=127.27.161.20:38599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260810 03:31:04.586776 25131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:31:04.575606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.161.1:45587 (local address 127.27.161.62:43637)
0810 03:31:04.576079 (+ 473us) server_negotiation.cc:206] Beginning negotiation
0810 03:31:04.576084 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:31:04.577291 (+ 1207us) server_negotiation.cc:407] Connection header received
0810 03:31:04.579638 (+ 2347us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:31:04.579644 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:31:04.579731 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:31:04.579832 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:31:04.581011 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:31:04.581785 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:31:04.584014 (+ 2229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:31:04.584415 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:31:04.584515 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:31:04.586026 (+ 1511us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:31:04.586031 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:31:04.586034 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:31:04.586058 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:31:04.586101 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:31:04.586114 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:31:04.586183 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0810 03:31:04.586418 (+ 235us) server_negotiation.cc:299] Negotiation successful
0810 03:31:04.586511 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358,"thread_start_us":223,"threads_started":1}
I20260810 03:31:04.588080 25132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:31:04.576224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:43637 (local address 127.27.161.1:45587)
0810 03:31:04.577125 (+ 901us) negotiation.cc:107] Waiting for socket to connect
0810 03:31:04.577166 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0810 03:31:04.579361 (+ 2195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:31:04.580073 (+ 712us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:31:04.580081 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:31:04.580171 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:31:04.580809 (+ 638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:31:04.580842 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:31:04.581993 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:31:04.581998 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:31:04.583868 (+ 1870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:31:04.583885 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:31:04.584634 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:31:04.584641 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:31:04.584930 (+ 289us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:31:04.585637 (+ 707us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 03:31:04.585640 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:31:04.585707 (+ 67us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 03:31:04.585806 (+ 99us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:31:04.586203 (+ 397us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:31:04.586215 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:31:04.586222 (+ 7us) client_negotiation.cc:769] Sending connection context
0810 03:31:04.586405 (+ 183us) client_negotiation.cc:240] Negotiation successful
0810 03:31:04.586803 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":308,"threads_started":1}
I20260810 03:31:04.589561 25129 heartbeater.cc:347] Connected to a master server at 127.27.161.62:43637
I20260810 03:31:04.589910 25129 heartbeater.cc:464] Registering TS with master...
I20260810 03:31:04.590903 25129 heartbeater.cc:511] Master 127.27.161.62:43637 requested a full tablet report, sending...
I20260810 03:31:04.593170 24834 ts_manager.cc:194] Registered new tserver with Master: 44e811ef403a4ff99f2056d86168de29 (127.27.161.1:40327)
I20260810 03:31:04.594684 24834 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.161.1:45587
W20260810 03:31:04.939215 25133 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:31:04.939522 25133 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:31:04.939658 25133 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:31:04.947132 25133 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:31:04.947259 25133 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:31:04.947407 25133 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.161.2
I20260810 03:31:04.956986 25133 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.161.20:38599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786332336319544-28292-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786332336319544-28292-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.161.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786332336319544-28292-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.161.2
--webserver_port=0
--tserver_master_addrs=127.27.161.62:43637
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.161.2
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786332336319544-28292-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:31:04.958338 25133 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:31:04.960150 25133 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:31:04.973385 25142 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:31:04.973418 25145 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:31:04.973546 25141 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:31:04.974260 25133 server_base.cc:1034] running on GCE node
I20260810 03:31:04.974884 25133 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:31:04.975875 25133 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:31:04.977092 25133 hybrid_clock.cc:648] HybridClock initialized: now 1786332664977027 us; error 52 us; skew 500 ppm
I20260810 03:31:04.977424 25133 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 03:31:04.979707 25133 webserver.cc:460] Webserver started at http://127.27.161.2:39189/ using document root <none> and password file <none>
I20260810 03:31:04.980453 25133 fs_manager.cc:359] Metadata directory not provided
I20260810 03:31:04.980562 25133 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:31:04.980876 25133 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:31:04.983968 25133 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786332336319544-28292-0/minicluster-data/ts-1/data/instance:
uuid: "50f0830e0c434e4da5a1c064392a4d34"
format_stamp: "Formatted at 2026-08-10 03:31:04 on dist-test-slave-vpvm"
server_key: "3ecb30af8260fdc8e4f5a5399e8a67af"
server_key_iv: "ce56e44a084843593f5f08159442206a"
server_key_version: "encryptionkey@0"
I20260810 03:31:04.984817 25133 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786332336319544-28292-0/minicluster-data/ts-1/wal/instance:
uuid: "50f0830e0c434e4da5a1c064392a4d34"
format_stamp: "Formatted at 2026-08-10 03:31:04 on dist-test-slave-vpvm"
server_key: "3ecb30af8260fdc8e4f5a5399e8a67af"
server_key_iv: "ce56e44a084843593f5f08159442206a"
server_key_version: "encryptionkey@0"
I20260810 03:31:04.990693 25133 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.004s
I20260810 03:31:04.994346 25152 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:31:04.995666 25133 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 03:31:04.995827 25133 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786332336319544-28292-0/minicluster-data/ts-1/data,/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786332336319544-28292-0/minicluster-data/ts-1/wal
uuid: "50f0830e0c434e4da5a1c064392a4d34"
format_stamp: "Formatted at 2026-08-10 03:31:04 on dist-test-slave-vpvm"
server_key: "3ecb30af8260fdc8e4f5a5399e8a67af"
server_key_iv: "ce56e44a084843593f5f08159442206a"
server_key_version: "encryptionkey@0"
I20260810 03:31:04.996037 25133 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786332336319544-28292-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786332336319544-28292-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786332336319544-28292-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:31:05.040895 25133 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:31:05.041790 25133 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:31:05.042114 25133 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:31:05.043113 25133 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:31:05.044656 25133 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:31:05.044749 25133 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:31:05.044857 25133 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:31:05.044926 25133 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:31:05.075902 25133 rpc_server.cc:307] RPC server started. Bound to: 127.27.161.2:41217
I20260810 03:31:05.075949 25319 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.161.2:41217 every 8 connection(s)
I20260810 03:31:05.077363 25133 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786332336319544-28292-0/minicluster-data/ts-1/data/info.pb
I20260810 03:31:05.079241 28292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iBH5/build/asan/bin/kudu as pid 25133
I20260810 03:31:05.079401 28292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786332336319544-28292-0/minicluster-data/ts-1/wal/instance
I20260810 03:31:05.079902 28292 external_mini_cluster.cc:1428] Setting key 14e11a85a84ad7e2cedf8f13b4a04d85
I20260810 03:31:05.083253 28292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iBH5/build/asan/bin/kudu
/tmp/dist-test-task24iBH5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786332336319544-28292-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786332336319544-28292-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786332336319544-28292-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786332336319544-28292-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.161.3:0
--local_ip_for_outbound_sockets=127.27.161.3
--webserver_interface=127.27.161.3
--webserver_port=0
--tserver_master_addrs=127.27.161.62:43637
--builtin_ntp_servers=127.27.161.20:38599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260810 03:31:05.091559 25131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:31:05.080117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.161.2:60851 (local address 127.27.161.62:43637)
0810 03:31:05.080355 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0810 03:31:05.080362 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:31:05.081982 (+ 1620us) server_negotiation.cc:407] Connection header received
0810 03:31:05.083979 (+ 1997us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:31:05.083985 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:31:05.084070 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:31:05.084177 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:31:05.085560 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:31:05.086646 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:31:05.088556 (+ 1910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:31:05.089101 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:31:05.089233 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:31:05.090717 (+ 1484us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:31:05.090724 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:31:05.090728 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:31:05.090754 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:31:05.090791 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:31:05.090806 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:31:05.090887 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0810 03:31:05.091136 (+ 249us) server_negotiation.cc:299] Negotiation successful
0810 03:31:05.091249 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260810 03:31:05.092610 25325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:31:05.080725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:43637 (local address 127.27.161.2:60851)
0810 03:31:05.081787 (+ 1062us) negotiation.cc:107] Waiting for socket to connect
0810 03:31:05.081844 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0810 03:31:05.083681 (+ 1837us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:31:05.084456 (+ 775us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:31:05.084465 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:31:05.084680 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:31:05.085345 (+ 665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:31:05.085367 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:31:05.086837 (+ 1470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:31:05.086865 (+ 28us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:31:05.088361 (+ 1496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:31:05.088376 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:31:05.089348 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:31:05.089353 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:31:05.089601 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:31:05.090358 (+ 757us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 03:31:05.090361 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:31:05.090406 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 03:31:05.090502 (+ 96us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:31:05.090905 (+ 403us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:31:05.090915 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:31:05.090922 (+ 7us) client_negotiation.cc:769] Sending connection context
0810 03:31:05.091099 (+ 177us) client_negotiation.cc:240] Negotiation successful
0810 03:31:05.091445 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":540,"thread_start_us":317,"threads_started":1}
I20260810 03:31:05.094167 25321 heartbeater.cc:347] Connected to a master server at 127.27.161.62:43637
I20260810 03:31:05.094542 25321 heartbeater.cc:464] Registering TS with master...
I20260810 03:31:05.095534 25321 heartbeater.cc:511] Master 127.27.161.62:43637 requested a full tablet report, sending...
I20260810 03:31:05.097273 24834 ts_manager.cc:194] Registered new tserver with Master: 50f0830e0c434e4da5a1c064392a4d34 (127.27.161.2:41217)
I20260810 03:31:05.098165 24834 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.161.2:60851
W20260810 03:31:05.445895 25327 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:31:05.446369 25327 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:31:05.446522 25327 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:31:05.456640 25327 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:31:05.456782 25327 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:31:05.456966 25327 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.161.3
I20260810 03:31:05.471654 25327 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.161.20:38599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786332336319544-28292-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786332336319544-28292-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.161.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786332336319544-28292-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.161.3
--webserver_port=0
--tserver_master_addrs=127.27.161.62:43637
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.161.3
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786332336319544-28292-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:31:05.473584 25327 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:31:05.476171 25327 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:31:05.490711 25341 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:31:05.490733 25337 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:31:05.490756 25338 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:31:05.491902 25327 server_base.cc:1034] running on GCE node
I20260810 03:31:05.492506 25327 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:31:05.493501 25327 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:31:05.494765 25327 hybrid_clock.cc:648] HybridClock initialized: now 1786332665494705 us; error 51 us; skew 500 ppm
I20260810 03:31:05.495076 25327 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 03:31:05.497408 25327 webserver.cc:460] Webserver started at http://127.27.161.3:36637/ using document root <none> and password file <none>
I20260810 03:31:05.498351 25327 fs_manager.cc:359] Metadata directory not provided
I20260810 03:31:05.498458 25327 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:31:05.498760 25327 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:31:05.502218 25327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786332336319544-28292-0/minicluster-data/ts-2/data/instance:
uuid: "d308aaa32e284d2d826728356c08997b"
format_stamp: "Formatted at 2026-08-10 03:31:05 on dist-test-slave-vpvm"
server_key: "5ee74693671820f2cc8d9690b7a4b54d"
server_key_iv: "7b0b81e0932216cd896fa33375650f4e"
server_key_version: "encryptionkey@0"
I20260810 03:31:05.503182 25327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786332336319544-28292-0/minicluster-data/ts-2/wal/instance:
uuid: "d308aaa32e284d2d826728356c08997b"
format_stamp: "Formatted at 2026-08-10 03:31:05 on dist-test-slave-vpvm"
server_key: "5ee74693671820f2cc8d9690b7a4b54d"
server_key_iv: "7b0b81e0932216cd896fa33375650f4e"
server_key_version: "encryptionkey@0"
I20260810 03:31:05.509150 25327 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260810 03:31:05.512866 25347 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:31:05.514329 25327 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260810 03:31:05.514518 25327 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786332336319544-28292-0/minicluster-data/ts-2/data,/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786332336319544-28292-0/minicluster-data/ts-2/wal
uuid: "d308aaa32e284d2d826728356c08997b"
format_stamp: "Formatted at 2026-08-10 03:31:05 on dist-test-slave-vpvm"
server_key: "5ee74693671820f2cc8d9690b7a4b54d"
server_key_iv: "7b0b81e0932216cd896fa33375650f4e"
server_key_version: "encryptionkey@0"
I20260810 03:31:05.514743 25327 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786332336319544-28292-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786332336319544-28292-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786332336319544-28292-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:31:05.541639 25327 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:31:05.542665 25327 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:31:05.542999 25327 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:31:05.544227 25327 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:31:05.545878 25327 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:31:05.546025 25327 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:31:05.546154 25327 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:31:05.546245 25327 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:31:05.579275 25327 rpc_server.cc:307] RPC server started. Bound to: 127.27.161.3:43083
I20260810 03:31:05.579334 25528 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.161.3:43083 every 8 connection(s)
I20260810 03:31:05.580669 25327 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786332336319544-28292-0/minicluster-data/ts-2/data/info.pb
I20260810 03:31:05.590895 28292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iBH5/build/asan/bin/kudu as pid 25327
I20260810 03:31:05.591063 28292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786332336319544-28292-0/minicluster-data/ts-2/wal/instance
I20260810 03:31:05.591609 28292 external_mini_cluster.cc:1428] Setting key 74cd6cb94d320ad8e6a7bcba9d8e9f67
I20260810 03:31:05.594597 25131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:31:05.583496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.161.3:55565 (local address 127.27.161.62:43637)
0810 03:31:05.583725 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0810 03:31:05.583730 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:31:05.585273 (+ 1543us) server_negotiation.cc:407] Connection header received
0810 03:31:05.587276 (+ 2003us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:31:05.587281 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:31:05.587361 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:31:05.587459 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:31:05.588710 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:31:05.589478 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:31:05.591329 (+ 1851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:31:05.591938 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:31:05.592083 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:31:05.593665 (+ 1582us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:31:05.593672 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:31:05.593677 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:31:05.593699 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:31:05.593740 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:31:05.593757 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:31:05.593850 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0810 03:31:05.594134 (+ 284us) server_negotiation.cc:299] Negotiation successful
0810 03:31:05.594241 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260810 03:31:05.595672 25534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:31:05.584071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:43637 (local address 127.27.161.3:55565)
0810 03:31:05.585095 (+ 1024us) negotiation.cc:107] Waiting for socket to connect
0810 03:31:05.585136 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0810 03:31:05.586923 (+ 1787us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:31:05.587728 (+ 805us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:31:05.587737 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:31:05.587842 (+ 105us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:31:05.588495 (+ 653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:31:05.588515 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:31:05.589636 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:31:05.589642 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:31:05.591171 (+ 1529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:31:05.591185 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:31:05.592197 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:31:05.592206 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:31:05.592490 (+ 284us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:31:05.593309 (+ 819us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 03:31:05.593312 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:31:05.593370 (+ 58us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 03:31:05.593466 (+ 96us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:31:05.593867 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:31:05.593877 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:31:05.593884 (+ 7us) client_negotiation.cc:769] Sending connection context
0810 03:31:05.594095 (+ 211us) client_negotiation.cc:240] Negotiation successful
0810 03:31:05.594453 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":539,"thread_start_us":301,"threads_started":1}
I20260810 03:31:05.597007 25531 heartbeater.cc:347] Connected to a master server at 127.27.161.62:43637
I20260810 03:31:05.597338 25531 heartbeater.cc:464] Registering TS with master...
I20260810 03:31:05.597782 25129 heartbeater.cc:503] Master 127.27.161.62:43637 was elected leader, sending a full tablet report...
I20260810 03:31:05.598277 25531 heartbeater.cc:511] Master 127.27.161.62:43637 requested a full tablet report, sending...
I20260810 03:31:05.599936 24834 ts_manager.cc:194] Registered new tserver with Master: d308aaa32e284d2d826728356c08997b (127.27.161.3:43083)
I20260810 03:31:05.600762 24834 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.161.3:55565
I20260810 03:31:05.607470 28292 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 03:31:05.617789 25131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:31:05.611871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56774 (local address 127.27.161.62:43637)
0810 03:31:05.612152 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0810 03:31:05.612160 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 03:31:05.612420 (+ 260us) server_negotiation.cc:407] Connection header received
0810 03:31:05.612565 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:31:05.612569 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:31:05.612643 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:31:05.612738 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:31:05.613184 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:31:05.613912 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:31:05.615390 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:31:05.615845 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:31:05.615945 (+ 100us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:31:05.617094 (+ 1149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:31:05.617099 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:31:05.617102 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:31:05.617121 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:31:05.617161 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:31:05.617173 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:31:05.617295 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0810 03:31:05.617408 (+ 113us) server_negotiation.cc:299] Negotiation successful
0810 03:31:05.617459 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260810 03:31:05.622128 24834 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56774:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 03:31:05.629145 24834 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 03:31:05.671665 25556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:31:05.642196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.1:40327 (local address 127.0.0.1:59362)
0810 03:31:05.661138 (+ 18942us) negotiation.cc:107] Waiting for socket to connect
0810 03:31:05.661162 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0810 03:31:05.661332 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:31:05.663579 (+ 2247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:31:05.663584 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:31:05.663611 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:31:05.663989 (+ 378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:31:05.664003 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:31:05.665656 (+ 1653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:31:05.665661 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:31:05.668129 (+ 2468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:31:05.668147 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:31:05.670575 (+ 2428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:31:05.670582 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:31:05.670782 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:31:05.670890 (+ 108us) client_negotiation.cc:769] Sending connection context
0810 03:31:05.671043 (+ 153us) client_negotiation.cc:240] Negotiation successful
0810 03:31:05.671247 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18771,"mutex_wait_us":31,"thread_start_us":19222,"threads_started":1}
I20260810 03:31:05.671778 25561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:31:05.642964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47580 (local address 127.27.161.2:41217)
0810 03:31:05.662148 (+ 19184us) server_negotiation.cc:206] Beginning negotiation
0810 03:31:05.662157 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 03:31:05.663420 (+ 1263us) server_negotiation.cc:407] Connection header received
0810 03:31:05.663541 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:31:05.663548 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:31:05.663751 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:31:05.663917 (+ 166us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:31:05.664486 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:31:05.665741 (+ 1255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:31:05.668432 (+ 2691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:31:05.669933 (+ 1501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:31:05.670097 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:31:05.670365 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0810 03:31:05.671204 (+ 839us) server_negotiation.cc:299] Negotiation successful
0810 03:31:05.671388 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18959,"thread_start_us":18839,"threads_started":1}
I20260810 03:31:05.671968 25554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:31:05.641733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.3:43083 (local address 127.0.0.1:35882)
0810 03:31:05.654871 (+ 13138us) negotiation.cc:107] Waiting for socket to connect
0810 03:31:05.654934 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0810 03:31:05.655193 (+ 259us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:31:05.662323 (+ 7130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:31:05.662332 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:31:05.662372 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:31:05.662973 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:31:05.662992 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:31:05.665120 (+ 2128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:31:05.665125 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:31:05.668690 (+ 3565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:31:05.668709 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:31:05.670596 (+ 1887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:31:05.670603 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:31:05.670881 (+ 278us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:31:05.670943 (+ 62us) client_negotiation.cc:769] Sending connection context
0810 03:31:05.671015 (+ 72us) client_negotiation.cc:240] Negotiation successful
0810 03:31:05.671458 (+ 443us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12745,"mutex_wait_us":423,"thread_start_us":12233,"threads_started":1}
I20260810 03:31:05.672005 25555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:31:05.641864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35882 (local address 127.27.161.3:43083)
0810 03:31:05.661135 (+ 19271us) server_negotiation.cc:206] Beginning negotiation
0810 03:31:05.661146 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 03:31:05.661191 (+ 45us) server_negotiation.cc:407] Connection header received
0810 03:31:05.661316 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:31:05.661326 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:31:05.661632 (+ 306us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:31:05.661892 (+ 260us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:31:05.663274 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:31:05.664961 (+ 1687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:31:05.668888 (+ 3927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:31:05.669932 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:31:05.670114 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:31:05.670448 (+ 334us) server_negotiation.cc:1035] Waiting for connection context
0810 03:31:05.671211 (+ 763us) server_negotiation.cc:299] Negotiation successful
0810 03:31:05.671405 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19041,"thread_start_us":18840,"threads_started":1}
I20260810 03:31:05.672524 25559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:31:05.642398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.2:41217 (local address 127.0.0.1:47580)
0810 03:31:05.663111 (+ 20713us) negotiation.cc:107] Waiting for socket to connect
0810 03:31:05.663134 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0810 03:31:05.663278 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:31:05.663948 (+ 670us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:31:05.663952 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:31:05.663981 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:31:05.664324 (+ 343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:31:05.664333 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:31:05.665913 (+ 1580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:31:05.665918 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:31:05.668175 (+ 2257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:31:05.668188 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:31:05.670524 (+ 2336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:31:05.670532 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:31:05.670828 (+ 296us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:31:05.670955 (+ 127us) client_negotiation.cc:769] Sending connection context
0810 03:31:05.671024 (+ 69us) client_negotiation.cc:240] Negotiation successful
0810 03:31:05.672201 (+ 1177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20571,"mutex_wait_us":1128,"thread_start_us":20369,"threads_started":1}
I20260810 03:31:05.672685 25557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:31:05.642440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59362 (local address 127.27.161.1:40327)
0810 03:31:05.662640 (+ 20200us) server_negotiation.cc:206] Beginning negotiation
0810 03:31:05.662649 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 03:31:05.662684 (+ 35us) server_negotiation.cc:407] Connection header received
0810 03:31:05.662782 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:31:05.662790 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:31:05.663069 (+ 279us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:31:05.663300 (+ 231us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:31:05.664211 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:31:05.665496 (+ 1285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:31:05.668407 (+ 2911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:31:05.669946 (+ 1539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:31:05.670100 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:31:05.670404 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0810 03:31:05.672250 (+ 1846us) server_negotiation.cc:299] Negotiation successful
0810 03:31:05.672422 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20020,"thread_start_us":19896,"threads_started":1}
I20260810 03:31:05.676384 25033 tablet_service.cc:1467] Processing CreateTablet for tablet 499a5e40dd854d07a6da91bd362da89c (DEFAULT_TABLE table=test-table [id=af5eb1b6c6cc43e588d52834b6b5858d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:31:05.676369 25229 tablet_service.cc:1467] Processing CreateTablet for tablet 499a5e40dd854d07a6da91bd362da89c (DEFAULT_TABLE table=test-table [id=af5eb1b6c6cc43e588d52834b6b5858d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:31:05.676384 25429 tablet_service.cc:1467] Processing CreateTablet for tablet 499a5e40dd854d07a6da91bd362da89c (DEFAULT_TABLE table=test-table [id=af5eb1b6c6cc43e588d52834b6b5858d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:31:05.677984 25429 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 499a5e40dd854d07a6da91bd362da89c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:31:05.677976 25033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 499a5e40dd854d07a6da91bd362da89c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:31:05.677994 25229 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 499a5e40dd854d07a6da91bd362da89c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:31:05.690833 25562 tablet_bootstrap.cc:492] T 499a5e40dd854d07a6da91bd362da89c P 50f0830e0c434e4da5a1c064392a4d34: Bootstrap starting.
I20260810 03:31:05.692693 25563 tablet_bootstrap.cc:492] T 499a5e40dd854d07a6da91bd362da89c P d308aaa32e284d2d826728356c08997b: Bootstrap starting.
I20260810 03:31:05.694234 25564 tablet_bootstrap.cc:492] T 499a5e40dd854d07a6da91bd362da89c P 44e811ef403a4ff99f2056d86168de29: Bootstrap starting.
I20260810 03:31:05.696460 25562 tablet_bootstrap.cc:654] T 499a5e40dd854d07a6da91bd362da89c P 50f0830e0c434e4da5a1c064392a4d34: Neither blocks nor log segments found. Creating new log.
I20260810 03:31:05.696877 25563 tablet_bootstrap.cc:654] T 499a5e40dd854d07a6da91bd362da89c P d308aaa32e284d2d826728356c08997b: Neither blocks nor log segments found. Creating new log.
I20260810 03:31:05.697230 25564 tablet_bootstrap.cc:654] T 499a5e40dd854d07a6da91bd362da89c P 44e811ef403a4ff99f2056d86168de29: Neither blocks nor log segments found. Creating new log.
I20260810 03:31:05.697847 25562 log.cc:824] T 499a5e40dd854d07a6da91bd362da89c P 50f0830e0c434e4da5a1c064392a4d34: Log is configured to *not* fsync() on all Append() calls
I20260810 03:31:05.698077 25563 log.cc:824] T 499a5e40dd854d07a6da91bd362da89c P d308aaa32e284d2d826728356c08997b: Log is configured to *not* fsync() on all Append() calls
I20260810 03:31:05.698297 25564 log.cc:824] T 499a5e40dd854d07a6da91bd362da89c P 44e811ef403a4ff99f2056d86168de29: Log is configured to *not* fsync() on all Append() calls
I20260810 03:31:05.701298 25562 tablet_bootstrap.cc:492] T 499a5e40dd854d07a6da91bd362da89c P 50f0830e0c434e4da5a1c064392a4d34: No bootstrap required, opened a new log
I20260810 03:31:05.701447 25564 tablet_bootstrap.cc:492] T 499a5e40dd854d07a6da91bd362da89c P 44e811ef403a4ff99f2056d86168de29: No bootstrap required, opened a new log
I20260810 03:31:05.701597 25562 ts_tablet_manager.cc:1397] T 499a5e40dd854d07a6da91bd362da89c P 50f0830e0c434e4da5a1c064392a4d34: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20260810 03:31:05.701757 25564 ts_tablet_manager.cc:1397] T 499a5e40dd854d07a6da91bd362da89c P 44e811ef403a4ff99f2056d86168de29: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.005s
I20260810 03:31:05.702064 25563 tablet_bootstrap.cc:492] T 499a5e40dd854d07a6da91bd362da89c P d308aaa32e284d2d826728356c08997b: No bootstrap required, opened a new log
I20260810 03:31:05.702349 25563 ts_tablet_manager.cc:1397] T 499a5e40dd854d07a6da91bd362da89c P d308aaa32e284d2d826728356c08997b: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20260810 03:31:05.709698 25562 raft_consensus.cc:348] T 499a5e40dd854d07a6da91bd362da89c P 50f0830e0c434e4da5a1c064392a4d34 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d308aaa32e284d2d826728356c08997b" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 43083 } } peers { permanent_uuid: "44e811ef403a4ff99f2056d86168de29" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 40327 } } peers { permanent_uuid: "50f0830e0c434e4da5a1c064392a4d34" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 41217 } }
I20260810 03:31:05.710222 25562 raft_consensus.cc:374] T 499a5e40dd854d07a6da91bd362da89c P 50f0830e0c434e4da5a1c064392a4d34 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:31:05.710325 25562 raft_consensus.cc:729] T 499a5e40dd854d07a6da91bd362da89c P 50f0830e0c434e4da5a1c064392a4d34 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50f0830e0c434e4da5a1c064392a4d34, State: Initialized, Role: FOLLOWER
I20260810 03:31:05.710942 25562 consensus_queue.cc:260] T 499a5e40dd854d07a6da91bd362da89c P 50f0830e0c434e4da5a1c064392a4d34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d308aaa32e284d2d826728356c08997b" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 43083 } } peers { permanent_uuid: "44e811ef403a4ff99f2056d86168de29" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 40327 } } peers { permanent_uuid: "50f0830e0c434e4da5a1c064392a4d34" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 41217 } }
I20260810 03:31:05.712127 25321 heartbeater.cc:503] Master 127.27.161.62:43637 was elected leader, sending a full tablet report...
I20260810 03:31:05.712625 25562 ts_tablet_manager.cc:1428] T 499a5e40dd854d07a6da91bd362da89c P 50f0830e0c434e4da5a1c064392a4d34: Time spent starting tablet: real 0.011s user 0.012s sys 0.000s
I20260810 03:31:05.712565 25564 raft_consensus.cc:348] T 499a5e40dd854d07a6da91bd362da89c P 44e811ef403a4ff99f2056d86168de29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d308aaa32e284d2d826728356c08997b" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 43083 } } peers { permanent_uuid: "44e811ef403a4ff99f2056d86168de29" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 40327 } } peers { permanent_uuid: "50f0830e0c434e4da5a1c064392a4d34" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 41217 } }
I20260810 03:31:05.713028 25564 raft_consensus.cc:374] T 499a5e40dd854d07a6da91bd362da89c P 44e811ef403a4ff99f2056d86168de29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:31:05.713133 25564 raft_consensus.cc:729] T 499a5e40dd854d07a6da91bd362da89c P 44e811ef403a4ff99f2056d86168de29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44e811ef403a4ff99f2056d86168de29, State: Initialized, Role: FOLLOWER
I20260810 03:31:05.713765 25564 consensus_queue.cc:260] T 499a5e40dd854d07a6da91bd362da89c P 44e811ef403a4ff99f2056d86168de29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d308aaa32e284d2d826728356c08997b" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 43083 } } peers { permanent_uuid: "44e811ef403a4ff99f2056d86168de29" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 40327 } } peers { permanent_uuid: "50f0830e0c434e4da5a1c064392a4d34" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 41217 } }
I20260810 03:31:05.715534 25563 raft_consensus.cc:348] T 499a5e40dd854d07a6da91bd362da89c P d308aaa32e284d2d826728356c08997b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d308aaa32e284d2d826728356c08997b" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 43083 } } peers { permanent_uuid: "44e811ef403a4ff99f2056d86168de29" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 40327 } } peers { permanent_uuid: "50f0830e0c434e4da5a1c064392a4d34" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 41217 } }
I20260810 03:31:05.716038 25563 raft_consensus.cc:374] T 499a5e40dd854d07a6da91bd362da89c P d308aaa32e284d2d826728356c08997b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:31:05.716138 25563 raft_consensus.cc:729] T 499a5e40dd854d07a6da91bd362da89c P d308aaa32e284d2d826728356c08997b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d308aaa32e284d2d826728356c08997b, State: Initialized, Role: FOLLOWER
I20260810 03:31:05.716861 25563 consensus_queue.cc:260] T 499a5e40dd854d07a6da91bd362da89c P d308aaa32e284d2d826728356c08997b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d308aaa32e284d2d826728356c08997b" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 43083 } } peers { permanent_uuid: "44e811ef403a4ff99f2056d86168de29" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 40327 } } peers { permanent_uuid: "50f0830e0c434e4da5a1c064392a4d34" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 41217 } }
I20260810 03:31:05.719067 25563 ts_tablet_manager.cc:1428] T 499a5e40dd854d07a6da91bd362da89c P d308aaa32e284d2d826728356c08997b: Time spent starting tablet: real 0.017s user 0.013s sys 0.000s
I20260810 03:31:05.719234 25531 heartbeater.cc:503] Master 127.27.161.62:43637 was elected leader, sending a full tablet report...
I20260810 03:31:05.720604 25564 ts_tablet_manager.cc:1428] T 499a5e40dd854d07a6da91bd362da89c P 44e811ef403a4ff99f2056d86168de29: Time spent starting tablet: real 0.019s user 0.011s sys 0.009s
W20260810 03:31:05.825662 25130 tablet.cc:2369] T 499a5e40dd854d07a6da91bd362da89c P 44e811ef403a4ff99f2056d86168de29: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 03:31:05.827481 25573 raft_consensus.cc:482] T 499a5e40dd854d07a6da91bd362da89c P d308aaa32e284d2d826728356c08997b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 03:31:05.827807 25573 raft_consensus.cc:504] T 499a5e40dd854d07a6da91bd362da89c P d308aaa32e284d2d826728356c08997b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d308aaa32e284d2d826728356c08997b" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 43083 } } peers { permanent_uuid: "44e811ef403a4ff99f2056d86168de29" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 40327 } } peers { permanent_uuid: "50f0830e0c434e4da5a1c064392a4d34" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 41217 } }
W20260810 03:31:05.829528 25322 tablet.cc:2369] T 499a5e40dd854d07a6da91bd362da89c P 50f0830e0c434e4da5a1c064392a4d34: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 03:31:05.829596 25573 leader_election.cc:288] T 499a5e40dd854d07a6da91bd362da89c P d308aaa32e284d2d826728356c08997b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44e811ef403a4ff99f2056d86168de29 (127.27.161.1:40327), 50f0830e0c434e4da5a1c064392a4d34 (127.27.161.2:41217)
W20260810 03:31:05.832404 25532 tablet.cc:2369] T 499a5e40dd854d07a6da91bd362da89c P d308aaa32e284d2d826728356c08997b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 03:31:05.835914 25534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:31:05.829833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.1:40327 (local address 127.27.161.3:38283)
0810 03:31:05.830094 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0810 03:31:05.830119 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0810 03:31:05.830257 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:31:05.831032 (+ 775us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:31:05.831037 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:31:05.831069 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:31:05.831226 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:31:05.831234 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:31:05.832208 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:31:05.832213 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:31:05.834048 (+ 1835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:31:05.834064 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:31:05.835175 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:31:05.835180 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:31:05.835367 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:31:05.835393 (+ 26us) client_negotiation.cc:769] Sending connection context
0810 03:31:05.835472 (+ 79us) client_negotiation.cc:240] Negotiation successful
0810 03:31:05.835613 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95}
I20260810 03:31:05.835918 25557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:31:05.830472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.161.3:38283 (local address 127.27.161.1:40327)
0810 03:31:05.830688 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0810 03:31:05.830694 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:31:05.830714 (+ 20us) server_negotiation.cc:407] Connection header received
0810 03:31:05.830787 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:31:05.830791 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:31:05.830862 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:31:05.830943 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:31:05.831382 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:31:05.832078 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:31:05.834209 (+ 2131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:31:05.835023 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:31:05.835174 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:31:05.835252 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0810 03:31:05.835550 (+ 298us) server_negotiation.cc:299] Negotiation successful
0810 03:31:05.835683 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260810 03:31:05.836860 25561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:31:05.830502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.161.3:49531 (local address 127.27.161.2:41217)
0810 03:31:05.830688 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0810 03:31:05.830694 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:31:05.831260 (+ 566us) server_negotiation.cc:407] Connection header received
0810 03:31:05.831451 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:31:05.831455 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:31:05.831545 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:31:05.831635 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:31:05.832320 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:31:05.833334 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:31:05.835170 (+ 1836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:31:05.835921 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:31:05.836090 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:31:05.836159 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0810 03:31:05.836565 (+ 406us) server_negotiation.cc:299] Negotiation successful
0810 03:31:05.836674 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260810 03:31:05.836938 25575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:31:05.830184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.2:41217 (local address 127.27.161.3:49531)
0810 03:31:05.831130 (+ 946us) negotiation.cc:107] Waiting for socket to connect
0810 03:31:05.831147 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 03:31:05.831298 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:31:05.831664 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:31:05.831669 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:31:05.831695 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:31:05.832055 (+ 360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:31:05.832065 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:31:05.833571 (+ 1506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:31:05.833575 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:31:05.834974 (+ 1399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:31:05.834988 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:31:05.836273 (+ 1285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:31:05.836278 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:31:05.836406 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:31:05.836430 (+ 24us) client_negotiation.cc:769] Sending connection context
0810 03:31:05.836506 (+ 76us) client_negotiation.cc:240] Negotiation successful
0810 03:31:05.836649 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":813,"thread_start_us":570,"threads_started":1}
I20260810 03:31:05.837092 25064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499a5e40dd854d07a6da91bd362da89c" candidate_uuid: "d308aaa32e284d2d826728356c08997b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44e811ef403a4ff99f2056d86168de29" is_pre_election: true
I20260810 03:31:05.837736 25064 raft_consensus.cc:2454] T 499a5e40dd854d07a6da91bd362da89c P 44e811ef403a4ff99f2056d86168de29 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d308aaa32e284d2d826728356c08997b in term 0.
I20260810 03:31:05.837932 25257 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499a5e40dd854d07a6da91bd362da89c" candidate_uuid: "d308aaa32e284d2d826728356c08997b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50f0830e0c434e4da5a1c064392a4d34" is_pre_election: true
I20260810 03:31:05.838479 25257 raft_consensus.cc:2454] T 499a5e40dd854d07a6da91bd362da89c P 50f0830e0c434e4da5a1c064392a4d34 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d308aaa32e284d2d826728356c08997b in term 0.
I20260810 03:31:05.838614 25355 leader_election.cc:302] T 499a5e40dd854d07a6da91bd362da89c P d308aaa32e284d2d826728356c08997b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44e811ef403a4ff99f2056d86168de29, d308aaa32e284d2d826728356c08997b; no voters:
I20260810 03:31:05.839022 25573 raft_consensus.cc:2781] T 499a5e40dd854d07a6da91bd362da89c P d308aaa32e284d2d826728356c08997b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 03:31:05.839196 25573 raft_consensus.cc:482] T 499a5e40dd854d07a6da91bd362da89c P d308aaa32e284d2d826728356c08997b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 03:31:05.839305 25573 raft_consensus.cc:3037] T 499a5e40dd854d07a6da91bd362da89c P d308aaa32e284d2d826728356c08997b [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:31:05.841439 25573 raft_consensus.cc:504] T 499a5e40dd854d07a6da91bd362da89c P d308aaa32e284d2d826728356c08997b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d308aaa32e284d2d826728356c08997b" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 43083 } } peers { permanent_uuid: "44e811ef403a4ff99f2056d86168de29" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 40327 } } peers { permanent_uuid: "50f0830e0c434e4da5a1c064392a4d34" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 41217 } }
I20260810 03:31:05.842044 25573 leader_election.cc:288] T 499a5e40dd854d07a6da91bd362da89c P d308aaa32e284d2d826728356c08997b [CANDIDATE]: Term 1 election: Requested vote from peers 44e811ef403a4ff99f2056d86168de29 (127.27.161.1:40327), 50f0830e0c434e4da5a1c064392a4d34 (127.27.161.2:41217)
I20260810 03:31:05.842593 25064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499a5e40dd854d07a6da91bd362da89c" candidate_uuid: "d308aaa32e284d2d826728356c08997b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44e811ef403a4ff99f2056d86168de29"
I20260810 03:31:05.842594 25257 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499a5e40dd854d07a6da91bd362da89c" candidate_uuid: "d308aaa32e284d2d826728356c08997b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50f0830e0c434e4da5a1c064392a4d34"
I20260810 03:31:05.842787 25064 raft_consensus.cc:3037] T 499a5e40dd854d07a6da91bd362da89c P 44e811ef403a4ff99f2056d86168de29 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:31:05.842795 25257 raft_consensus.cc:3037] T 499a5e40dd854d07a6da91bd362da89c P 50f0830e0c434e4da5a1c064392a4d34 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:31:05.844960 25064 raft_consensus.cc:2454] T 499a5e40dd854d07a6da91bd362da89c P 44e811ef403a4ff99f2056d86168de29 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d308aaa32e284d2d826728356c08997b in term 1.
I20260810 03:31:05.845201 25257 raft_consensus.cc:2454] T 499a5e40dd854d07a6da91bd362da89c P 50f0830e0c434e4da5a1c064392a4d34 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d308aaa32e284d2d826728356c08997b in term 1.
I20260810 03:31:05.845394 25355 leader_election.cc:302] T 499a5e40dd854d07a6da91bd362da89c P d308aaa32e284d2d826728356c08997b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44e811ef403a4ff99f2056d86168de29, d308aaa32e284d2d826728356c08997b; no voters:
I20260810 03:31:05.845719 25573 raft_consensus.cc:2781] T 499a5e40dd854d07a6da91bd362da89c P d308aaa32e284d2d826728356c08997b [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:31:05.846200 25573 raft_consensus.cc:686] T 499a5e40dd854d07a6da91bd362da89c P d308aaa32e284d2d826728356c08997b [term 1 LEADER]: Becoming Leader. State: Replica: d308aaa32e284d2d826728356c08997b, State: Running, Role: LEADER
I20260810 03:31:05.846661 25573 consensus_queue.cc:237] T 499a5e40dd854d07a6da91bd362da89c P d308aaa32e284d2d826728356c08997b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d308aaa32e284d2d826728356c08997b" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 43083 } } peers { permanent_uuid: "44e811ef403a4ff99f2056d86168de29" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 40327 } } peers { permanent_uuid: "50f0830e0c434e4da5a1c064392a4d34" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 41217 } }
I20260810 03:31:05.851079 24832 catalog_manager.cc:5495] T 499a5e40dd854d07a6da91bd362da89c P d308aaa32e284d2d826728356c08997b reported cstate change: term changed from 0 to 1, leader changed from <none> to d308aaa32e284d2d826728356c08997b (127.27.161.3). New cstate: current_term: 1 leader_uuid: "d308aaa32e284d2d826728356c08997b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d308aaa32e284d2d826728356c08997b" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 43083 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "44e811ef403a4ff99f2056d86168de29" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 40327 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "50f0830e0c434e4da5a1c064392a4d34" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 41217 } health_report { overall_health: UNKNOWN } } }
I20260810 03:31:05.868047 25555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:31:05.862737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35892 (local address 127.27.161.3:43083)
0810 03:31:05.862953 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0810 03:31:05.862958 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:31:05.862999 (+ 41us) server_negotiation.cc:407] Connection header received
0810 03:31:05.863204 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:31:05.863207 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:31:05.863309 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:31:05.863435 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 03:31:05.863849 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:31:05.864540 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:31:05.865883 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:31:05.866320 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:31:05.866416 (+ 96us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:31:05.866933 (+ 517us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 03:31:05.867440 (+ 507us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 03:31:05.867571 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0810 03:31:05.867745 (+ 174us) server_negotiation.cc:299] Negotiation successful
0810 03:31:05.867787 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260810 03:31:05.876890 25257 raft_consensus.cc:1261] T 499a5e40dd854d07a6da91bd362da89c P 50f0830e0c434e4da5a1c064392a4d34 [term 1 FOLLOWER]: Refusing update from remote peer d308aaa32e284d2d826728356c08997b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:31:05.877012 25064 raft_consensus.cc:1261] T 499a5e40dd854d07a6da91bd362da89c P 44e811ef403a4ff99f2056d86168de29 [term 1 FOLLOWER]: Refusing update from remote peer d308aaa32e284d2d826728356c08997b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:31:05.877997 25576 consensus_queue.cc:1035] T 499a5e40dd854d07a6da91bd362da89c P d308aaa32e284d2d826728356c08997b [LEADER]: Connected to new peer: Peer: permanent_uuid: "44e811ef403a4ff99f2056d86168de29" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 40327 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:31:05.878368 25573 consensus_queue.cc:1035] T 499a5e40dd854d07a6da91bd362da89c P d308aaa32e284d2d826728356c08997b [LEADER]: Connected to new peer: Peer: permanent_uuid: "50f0830e0c434e4da5a1c064392a4d34" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 41217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:31:05.891319 25580 mvcc.cc:205] Tried to move back new op lower bound from 7316818599421280256 to 7316818599311507456. Current Snapshot: MvccSnapshot[applied={T|T < 7316818599421280256}]
I20260810 03:31:05.898192 25582 mvcc.cc:205] Tried to move back new op lower bound from 7316818599421280256 to 7316818599311507456. Current Snapshot: MvccSnapshot[applied={T|T < 7316818599421280256}]
I20260810 03:31:05.906992 24832 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56774:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 03:31:05.907323 24832 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56774:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 03:31:05.911936 24832 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 306967a259714361bb8308b13373dc85: Sending DeleteTablet for 3 replicas of tablet 499a5e40dd854d07a6da91bd362da89c
I20260810 03:31:05.913275 25429 tablet_service.cc:1514] Processing DeleteTablet for tablet 499a5e40dd854d07a6da91bd362da89c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:31:05 UTC) from {username='slave'} at 127.0.0.1:35882
I20260810 03:31:05.913357 25229 tablet_service.cc:1514] Processing DeleteTablet for tablet 499a5e40dd854d07a6da91bd362da89c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:31:05 UTC) from {username='slave'} at 127.0.0.1:47580
I20260810 03:31:05.914215 25033 tablet_service.cc:1514] Processing DeleteTablet for tablet 499a5e40dd854d07a6da91bd362da89c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:31:05 UTC) from {username='slave'} at 127.0.0.1:59362
I20260810 03:31:05.914371 25593 tablet_replica.cc:330] T 499a5e40dd854d07a6da91bd362da89c P 50f0830e0c434e4da5a1c064392a4d34: stopping tablet replica
I20260810 03:31:05.914860 25593 raft_consensus.cc:2229] T 499a5e40dd854d07a6da91bd362da89c P 50f0830e0c434e4da5a1c064392a4d34 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 03:31:05.915441 25593 raft_consensus.cc:2258] T 499a5e40dd854d07a6da91bd362da89c P 50f0830e0c434e4da5a1c064392a4d34 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:31:05.916615 28292 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iBH5/build/asan/bin/kudu pid 24926
I20260810 03:31:05.917984 25593 ts_tablet_manager.cc:1905] T 499a5e40dd854d07a6da91bd362da89c P 50f0830e0c434e4da5a1c064392a4d34: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:31:05.925920 25593 ts_tablet_manager.cc:1918] T 499a5e40dd854d07a6da91bd362da89c P 50f0830e0c434e4da5a1c064392a4d34: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 03:31:05.926132 25593 log.cc:1196] T 499a5e40dd854d07a6da91bd362da89c P 50f0830e0c434e4da5a1c064392a4d34: Deleting WAL directory at /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786332336319544-28292-0/minicluster-data/ts-1/wal/wals/499a5e40dd854d07a6da91bd362da89c
I20260810 03:31:05.926743 25593 ts_tablet_manager.cc:1939] T 499a5e40dd854d07a6da91bd362da89c P 50f0830e0c434e4da5a1c064392a4d34: Deleting consensus metadata
I20260810 03:31:05.928897 24807 catalog_manager.cc:4841] TS 50f0830e0c434e4da5a1c064392a4d34 (127.27.161.2:41217): tablet 499a5e40dd854d07a6da91bd362da89c (table test-table [id=af5eb1b6c6cc43e588d52834b6b5858d]) successfully deleted
I20260810 03:31:05.929378 25592 tablet_replica.cc:330] T 499a5e40dd854d07a6da91bd362da89c P d308aaa32e284d2d826728356c08997b: stopping tablet replica
I20260810 03:31:05.929936 25592 raft_consensus.cc:2229] T 499a5e40dd854d07a6da91bd362da89c P d308aaa32e284d2d826728356c08997b [term 1 LEADER]: Raft consensus shutting down.
I20260810 03:31:05.930739 25592 raft_consensus.cc:2258] T 499a5e40dd854d07a6da91bd362da89c P d308aaa32e284d2d826728356c08997b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:31:05.933243 25592 ts_tablet_manager.cc:1905] T 499a5e40dd854d07a6da91bd362da89c P d308aaa32e284d2d826728356c08997b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:31:05.939404 25595 tablet_replica.cc:330] T 499a5e40dd854d07a6da91bd362da89c P 44e811ef403a4ff99f2056d86168de29: stopping tablet replica
I20260810 03:31:05.939975 25595 raft_consensus.cc:2229] T 499a5e40dd854d07a6da91bd362da89c P 44e811ef403a4ff99f2056d86168de29 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 03:31:05.940601 25595 raft_consensus.cc:2258] T 499a5e40dd854d07a6da91bd362da89c P 44e811ef403a4ff99f2056d86168de29 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:31:05.942898 25595 ts_tablet_manager.cc:1905] T 499a5e40dd854d07a6da91bd362da89c P 44e811ef403a4ff99f2056d86168de29: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:31:05.947127 25557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:31:05.918278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59368 (local address 127.27.161.1:40327)
0810 03:31:05.918511 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0810 03:31:05.918518 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:31:05.938667 (+ 20149us) server_negotiation.cc:407] Connection header received
0810 03:31:05.938878 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:31:05.938890 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:31:05.939040 (+ 150us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:31:05.939191 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:31:05.939874 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:31:05.941157 (+ 1283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:31:05.943201 (+ 2044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:31:05.943869 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:31:05.944040 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:31:05.945298 (+ 1258us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:31:05.945311 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:31:05.945330 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:31:05.946162 (+ 832us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:31:05.946226 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:31:05.946253 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:31:05.946344 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0810 03:31:05.946616 (+ 272us) server_negotiation.cc:299] Negotiation successful
0810 03:31:05.946741 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260810 03:31:05.950573 25592 ts_tablet_manager.cc:1918] T 499a5e40dd854d07a6da91bd362da89c P d308aaa32e284d2d826728356c08997b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 03:31:05.950793 25592 log.cc:1196] T 499a5e40dd854d07a6da91bd362da89c P d308aaa32e284d2d826728356c08997b: Deleting WAL directory at /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786332336319544-28292-0/minicluster-data/ts-2/wal/wals/499a5e40dd854d07a6da91bd362da89c
I20260810 03:31:05.951370 25592 ts_tablet_manager.cc:1939] T 499a5e40dd854d07a6da91bd362da89c P d308aaa32e284d2d826728356c08997b: Deleting consensus metadata
I20260810 03:31:05.951975 25595 ts_tablet_manager.cc:1918] T 499a5e40dd854d07a6da91bd362da89c P 44e811ef403a4ff99f2056d86168de29: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 03:31:05.952165 25595 log.cc:1196] T 499a5e40dd854d07a6da91bd362da89c P 44e811ef403a4ff99f2056d86168de29: Deleting WAL directory at /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786332336319544-28292-0/minicluster-data/ts-0/wal/wals/499a5e40dd854d07a6da91bd362da89c
I20260810 03:31:05.952737 25595 ts_tablet_manager.cc:1939] T 499a5e40dd854d07a6da91bd362da89c P 44e811ef403a4ff99f2056d86168de29: Deleting consensus metadata
I20260810 03:31:05.953414 24806 catalog_manager.cc:4841] TS d308aaa32e284d2d826728356c08997b (127.27.161.3:43083): tablet 499a5e40dd854d07a6da91bd362da89c (table test-table [id=af5eb1b6c6cc43e588d52834b6b5858d]) successfully deleted
I20260810 03:31:05.954792 25120 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:31:05.954731 24815 catalog_manager.cc:4841] TS 44e811ef403a4ff99f2056d86168de29 (127.27.161.1:40327): tablet 499a5e40dd854d07a6da91bd362da89c (table test-table [id=af5eb1b6c6cc43e588d52834b6b5858d]) successfully deleted
I20260810 03:31:06.095299 28292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iBH5/build/asan/bin/kudu with pid 24926
I20260810 03:31:06.131189 28292 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iBH5/build/asan/bin/kudu pid 25133
I20260810 03:31:06.138878 25561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:31:06.132075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47594 (local address 127.27.161.2:41217)
0810 03:31:06.132251 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0810 03:31:06.132256 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:31:06.132276 (+ 20us) server_negotiation.cc:407] Connection header received
0810 03:31:06.132358 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:31:06.132362 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:31:06.132449 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:31:06.132532 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:31:06.132975 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:31:06.133779 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:31:06.135605 (+ 1826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:31:06.136066 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:31:06.136181 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:31:06.137373 (+ 1192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:31:06.137382 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:31:06.137398 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:31:06.138098 (+ 700us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:31:06.138155 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:31:06.138180 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:31:06.138269 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0810 03:31:06.138420 (+ 151us) server_negotiation.cc:299] Negotiation successful
0810 03:31:06.138524 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260810 03:31:06.139680 25313 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:31:06.229857 28292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iBH5/build/asan/bin/kudu with pid 25133
I20260810 03:31:06.267153 28292 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iBH5/build/asan/bin/kudu pid 25327
I20260810 03:31:06.279256 25555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:31:06.268029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35900 (local address 127.27.161.3:43083)
0810 03:31:06.268217 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0810 03:31:06.268223 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:31:06.268286 (+ 63us) server_negotiation.cc:407] Connection header received
0810 03:31:06.268500 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:31:06.268507 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:31:06.268637 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:31:06.268730 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:31:06.269191 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:31:06.270091 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:31:06.271437 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:31:06.271917 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:31:06.272045 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:31:06.277581 (+ 5536us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:31:06.277603 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:31:06.277621 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:31:06.278380 (+ 759us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:31:06.278437 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:31:06.278465 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:31:06.278538 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0810 03:31:06.278786 (+ 248us) server_negotiation.cc:299] Negotiation successful
0810 03:31:06.278906 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260810 03:31:06.280069 25521 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:31:06.404246 28292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iBH5/build/asan/bin/kudu with pid 25327
I20260810 03:31:06.427693 28292 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iBH5/build/asan/bin/kudu pid 24786
I20260810 03:31:06.428891 24880 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:31:06.539819 28292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iBH5/build/asan/bin/kudu with pid 24786
2026-08-10T03:31:06Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (3161 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786332336319544-28292-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 03:31:06 dist-test-slave-vpvm krb5kdc[25604](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 03:31:06 dist-test-slave-vpvm krb5kdc[25604](info): set up 2 sockets
Aug 10 03:31:06 dist-test-slave-vpvm krb5kdc[25604](info): commencing operation
krb5kdc: starting...
W20260810 03:31:07.806238 28292 mini_kdc.cc:121] Time spent starting KDC: real 1.217s user 0.005s sys 0.052s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786332336319544-28292-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786332336319544-28292-0/krb5kdc/test-user.keytab.
Aug 10 03:31:08 dist-test-slave-vpvm krb5kdc[25604](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332668, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T03:31:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T03:31:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.161.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.161.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.161.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.62.keytab.
Entry for principal kudu/127.27.161.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.62.keytab.
I20260810 03:31:08.157469 28292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iBH5/build/asan/bin/kudu
/tmp/dist-test-task24iBH5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786332336319544-28292-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786332336319544-28292-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786332336319544-28292-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786332336319544-28292-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.161.62:43583
--webserver_interface=127.27.161.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.62.keytab
--principal=kudu/127.27.161.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.161.20:40853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.161.62:43583
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786332336319544-28292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786332336319544-28292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786332336319544-28292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:31:08.519428 25624 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:31:08.519764 25624 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:31:08.519901 25624 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:31:08.527336 25624 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:31:08.527464 25624 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:31:08.527524 25624 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:31:08.527604 25624 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:31:08.527663 25624 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:31:08.537750 25624 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.161.20:40853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786332336319544-28292-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786332336319544-28292-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.161.62:43583
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.62.keytab
--principal=kudu/127.27.161.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.161.62:43583
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786332336319544-28292-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.161.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786332336319544-28292-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:31:08.539139 25624 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:31:08.540483 25624 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:31:08.552382 25630 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:31:08.552317 25631 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:31:08.552353 25633 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:31:08.553512 25624 server_base.cc:1034] running on GCE node
I20260810 03:31:08.554355 25624 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:31:08.555799 25624 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:31:08.557008 25624 hybrid_clock.cc:648] HybridClock initialized: now 1786332668556956 us; error 42 us; skew 500 ppm
I20260810 03:31:08.557360 25624 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:31:08 dist-test-slave-vpvm krb5kdc[25604](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332668, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.161.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:31:08.560447 25624 init.cc:377] Logged in from keytab as kudu/127.27.161.62@KRBTEST.COM (short username kudu)
I20260810 03:31:08.562001 25624 webserver.cc:460] Webserver started at http://127.27.161.62:34159/ using document root <none> and password file <none>
I20260810 03:31:08.562726 25624 fs_manager.cc:359] Metadata directory not provided
I20260810 03:31:08.562851 25624 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:31:08.563148 25624 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:31:08.566314 25624 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786332336319544-28292-0/minicluster-data/master-0/data/instance:
uuid: "d4e6f3c481b6439aa7b096a70a12964a"
format_stamp: "Formatted at 2026-08-10 03:31:08 on dist-test-slave-vpvm"
server_key: "5e855d9ee5e1b30a2c7dbcd61a153dc0"
server_key_iv: "8d3c0351931b5554986f243bc0ad5aa0"
server_key_version: "encryptionkey@0"
I20260810 03:31:08.567056 25624 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786332336319544-28292-0/minicluster-data/master-0/wal/instance:
uuid: "d4e6f3c481b6439aa7b096a70a12964a"
format_stamp: "Formatted at 2026-08-10 03:31:08 on dist-test-slave-vpvm"
server_key: "5e855d9ee5e1b30a2c7dbcd61a153dc0"
server_key_iv: "8d3c0351931b5554986f243bc0ad5aa0"
server_key_version: "encryptionkey@0"
I20260810 03:31:08.572325 25624 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260810 03:31:08.575928 25643 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:31:08.577227 25624 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 03:31:08.577394 25624 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786332336319544-28292-0/minicluster-data/master-0/data,/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786332336319544-28292-0/minicluster-data/master-0/wal
uuid: "d4e6f3c481b6439aa7b096a70a12964a"
format_stamp: "Formatted at 2026-08-10 03:31:08 on dist-test-slave-vpvm"
server_key: "5e855d9ee5e1b30a2c7dbcd61a153dc0"
server_key_iv: "8d3c0351931b5554986f243bc0ad5aa0"
server_key_version: "encryptionkey@0"
I20260810 03:31:08.577577 25624 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786332336319544-28292-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786332336319544-28292-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786332336319544-28292-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:31:08.624531 25624 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:31:08.629101 25624 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:31:08.629408 25624 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:31:08.647166 25624 rpc_server.cc:307] RPC server started. Bound to: 127.27.161.62:43583
I20260810 03:31:08.647189 25725 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.161.62:43583 every 8 connection(s)
I20260810 03:31:08.648466 25624 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786332336319544-28292-0/minicluster-data/master-0/data/info.pb
I20260810 03:31:08.652154 25726 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:31:08.653478 28292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iBH5/build/asan/bin/kudu as pid 25624
I20260810 03:31:08.653735 28292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786332336319544-28292-0/minicluster-data/master-0/wal/instance
I20260810 03:31:08.654327 28292 external_mini_cluster.cc:1428] Setting key 74af77b4cfcb9920065796fc303f17ea
I20260810 03:31:08.666270 25726 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d4e6f3c481b6439aa7b096a70a12964a: Bootstrap starting.
Aug 10 03:31:08 dist-test-slave-vpvm krb5kdc[25604](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332668, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.161.62@KRBTEST.COM
I20260810 03:31:08.670257 25726 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d4e6f3c481b6439aa7b096a70a12964a: Neither blocks nor log segments found. Creating new log.
I20260810 03:31:08.671432 25726 log.cc:824] T 00000000000000000000000000000000 P d4e6f3c481b6439aa7b096a70a12964a: Log is configured to *not* fsync() on all Append() calls
I20260810 03:31:08.674837 25726 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d4e6f3c481b6439aa7b096a70a12964a: No bootstrap required, opened a new log
I20260810 03:31:08.678565 25732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:31:08.656283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47196 (local address 127.27.161.62:43583)
0810 03:31:08.657134 (+ 851us) server_negotiation.cc:206] Beginning negotiation
0810 03:31:08.657146 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 03:31:08.657192 (+ 46us) server_negotiation.cc:407] Connection header received
0810 03:31:08.658986 (+ 1794us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:31:08.659015 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:31:08.659425 (+ 410us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:31:08.659763 (+ 338us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:31:08.661674 (+ 1911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:31:08.663101 (+ 1427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:31:08.665002 (+ 1901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:31:08.665775 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:31:08.665940 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:31:08.670884 (+ 4944us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:31:08.670892 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:31:08.670908 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:31:08.670952 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:31:08.674157 (+ 3205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:31:08.675057 (+ 900us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:31:08.675063 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:31:08.675070 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:31:08.675258 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:31:08.675721 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:31:08.675724 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:31:08.675726 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:31:08.676267 (+ 541us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:31:08.676481 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0810 03:31:08.676989 (+ 508us) server_negotiation.cc:299] Negotiation successful
0810 03:31:08.677398 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":623,"thread_start_us":378,"threads_started":1}
I20260810 03:31:08.683772 25726 raft_consensus.cc:348] T 00000000000000000000000000000000 P d4e6f3c481b6439aa7b096a70a12964a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4e6f3c481b6439aa7b096a70a12964a" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 43583 } }
I20260810 03:31:08.684206 25726 raft_consensus.cc:374] T 00000000000000000000000000000000 P d4e6f3c481b6439aa7b096a70a12964a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:31:08.684331 25726 raft_consensus.cc:729] T 00000000000000000000000000000000 P d4e6f3c481b6439aa7b096a70a12964a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4e6f3c481b6439aa7b096a70a12964a, State: Initialized, Role: FOLLOWER
I20260810 03:31:08.685000 25726 consensus_queue.cc:260] T 00000000000000000000000000000000 P d4e6f3c481b6439aa7b096a70a12964a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4e6f3c481b6439aa7b096a70a12964a" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 43583 } }
I20260810 03:31:08.685247 25726 raft_consensus.cc:388] T 00000000000000000000000000000000 P d4e6f3c481b6439aa7b096a70a12964a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 03:31:08.685359 25726 raft_consensus.cc:482] T 00000000000000000000000000000000 P d4e6f3c481b6439aa7b096a70a12964a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 03:31:08.685496 25726 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d4e6f3c481b6439aa7b096a70a12964a [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:31:08.687739 25726 raft_consensus.cc:504] T 00000000000000000000000000000000 P d4e6f3c481b6439aa7b096a70a12964a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4e6f3c481b6439aa7b096a70a12964a" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 43583 } }
I20260810 03:31:08.688253 25726 leader_election.cc:302] T 00000000000000000000000000000000 P d4e6f3c481b6439aa7b096a70a12964a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d4e6f3c481b6439aa7b096a70a12964a; no voters:
I20260810 03:31:08.688824 25726 leader_election.cc:288] T 00000000000000000000000000000000 P d4e6f3c481b6439aa7b096a70a12964a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 03:31:08.688962 25736 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d4e6f3c481b6439aa7b096a70a12964a [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:31:08.689642 25736 raft_consensus.cc:686] T 00000000000000000000000000000000 P d4e6f3c481b6439aa7b096a70a12964a [term 1 LEADER]: Becoming Leader. State: Replica: d4e6f3c481b6439aa7b096a70a12964a, State: Running, Role: LEADER
I20260810 03:31:08.690315 25736 consensus_queue.cc:237] T 00000000000000000000000000000000 P d4e6f3c481b6439aa7b096a70a12964a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4e6f3c481b6439aa7b096a70a12964a" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 43583 } }
I20260810 03:31:08.691088 25726 sys_catalog.cc:564] T 00000000000000000000000000000000 P d4e6f3c481b6439aa7b096a70a12964a [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:31:08.693776 25738 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4e6f3c481b6439aa7b096a70a12964a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d4e6f3c481b6439aa7b096a70a12964a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4e6f3c481b6439aa7b096a70a12964a" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 43583 } } }
I20260810 03:31:08.694363 25738 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4e6f3c481b6439aa7b096a70a12964a [sys.catalog]: This master's current role is: LEADER
I20260810 03:31:08.693756 25739 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4e6f3c481b6439aa7b096a70a12964a [sys.catalog]: SysCatalogTable state changed. Reason: New leader d4e6f3c481b6439aa7b096a70a12964a. Latest consensus state: current_term: 1 leader_uuid: "d4e6f3c481b6439aa7b096a70a12964a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4e6f3c481b6439aa7b096a70a12964a" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 43583 } } }
I20260810 03:31:08.694792 25739 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4e6f3c481b6439aa7b096a70a12964a [sys.catalog]: This master's current role is: LEADER
I20260810 03:31:08.699386 25744 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 03:31:08.704813 25744 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 03:31:08.714775 25744 catalog_manager.cc:1347] Generated new cluster ID: 5e3b5df33b9d4877b810246dec50266c
I20260810 03:31:08.714912 25744 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 03:31:08.742605 25744 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 03:31:08.744544 25744 catalog_manager.cc:1504] Loading token signing keys...
I20260810 03:31:08.754276 25744 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d4e6f3c481b6439aa7b096a70a12964a: Generated new TSK 0
I20260810 03:31:08.755396 25744 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.161.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.161.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.161.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.1.keytab.
Entry for principal kudu/127.27.161.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.1.keytab.
I20260810 03:31:08.834820 28292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iBH5/build/asan/bin/kudu
/tmp/dist-test-task24iBH5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786332336319544-28292-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786332336319544-28292-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786332336319544-28292-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786332336319544-28292-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.161.1:0
--local_ip_for_outbound_sockets=127.27.161.1
--webserver_interface=127.27.161.1
--webserver_port=0
--tserver_master_addrs=127.27.161.62:43583
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.1.keytab
--principal=kudu/127.27.161.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.161.20:40853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786332336319544-28292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786332336319544-28292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786332336319544-28292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:31:09.186196 25768 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:31:09.186501 25768 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:31:09.186606 25768 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:31:09.194485 25768 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:31:09.194620 25768 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:31:09.194747 25768 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.161.1
I20260810 03:31:09.205072 25768 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.161.20:40853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786332336319544-28292-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786332336319544-28292-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.1.keytab
--principal=kudu/127.27.161.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.161.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786332336319544-28292-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.161.1
--webserver_port=0
--tserver_master_addrs=127.27.161.62:43583
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.161.1
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786332336319544-28292-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:31:09.206770 25768 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:31:09.208199 25768 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:31:09.221652 25776 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:31:09.221633 25783 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:31:09.222513 25775 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:31:09.222882 25768 server_base.cc:1034] running on GCE node
I20260810 03:31:09.223524 25768 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:31:09.224551 25768 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:31:09.225787 25768 hybrid_clock.cc:648] HybridClock initialized: now 1786332669225732 us; error 43 us; skew 500 ppm
I20260810 03:31:09.226174 25768 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:31:09 dist-test-slave-vpvm krb5kdc[25604](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332669, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.161.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:31:09.229362 25768 init.cc:377] Logged in from keytab as kudu/127.27.161.1@KRBTEST.COM (short username kudu)
I20260810 03:31:09.230909 25768 webserver.cc:460] Webserver started at http://127.27.161.1:46523/ using document root <none> and password file <none>
I20260810 03:31:09.231582 25768 fs_manager.cc:359] Metadata directory not provided
I20260810 03:31:09.231686 25768 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:31:09.231956 25768 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:31:09.234942 25768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786332336319544-28292-0/minicluster-data/ts-0/data/instance:
uuid: "249100e6cc06451080ba73097f4b75c4"
format_stamp: "Formatted at 2026-08-10 03:31:09 on dist-test-slave-vpvm"
server_key: "df21fd0733371670e1f4204a880acf7f"
server_key_iv: "07ebcd10f748c9322f0118384bc631b4"
server_key_version: "encryptionkey@0"
I20260810 03:31:09.235678 25768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786332336319544-28292-0/minicluster-data/ts-0/wal/instance:
uuid: "249100e6cc06451080ba73097f4b75c4"
format_stamp: "Formatted at 2026-08-10 03:31:09 on dist-test-slave-vpvm"
server_key: "df21fd0733371670e1f4204a880acf7f"
server_key_iv: "07ebcd10f748c9322f0118384bc631b4"
server_key_version: "encryptionkey@0"
I20260810 03:31:09.240991 25768 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260810 03:31:09.244573 25796 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:31:09.245918 25768 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 03:31:09.246143 25768 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786332336319544-28292-0/minicluster-data/ts-0/data,/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786332336319544-28292-0/minicluster-data/ts-0/wal
uuid: "249100e6cc06451080ba73097f4b75c4"
format_stamp: "Formatted at 2026-08-10 03:31:09 on dist-test-slave-vpvm"
server_key: "df21fd0733371670e1f4204a880acf7f"
server_key_iv: "07ebcd10f748c9322f0118384bc631b4"
server_key_version: "encryptionkey@0"
I20260810 03:31:09.246338 25768 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786332336319544-28292-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786332336319544-28292-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786332336319544-28292-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:31:09.293731 25768 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:31:09.297706 25768 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:31:09.297976 25768 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:31:09.298997 25768 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:31:09.300339 25768 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:31:09.300414 25768 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:31:09.300510 25768 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:31:09.300554 25768 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:31:09.332232 25768 rpc_server.cc:307] RPC server started. Bound to: 127.27.161.1:35683
I20260810 03:31:09.332293 25976 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.161.1:35683 every 8 connection(s)
I20260810 03:31:09.333508 25768 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786332336319544-28292-0/minicluster-data/ts-0/data/info.pb
I20260810 03:31:09.339202 28292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iBH5/build/asan/bin/kudu as pid 25768
I20260810 03:31:09.339361 28292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786332336319544-28292-0/minicluster-data/ts-0/wal/instance
I20260810 03:31:09.339815 28292 external_mini_cluster.cc:1428] Setting key f50bd72d191d3c5acbde0a60a220e555
Aug 10 03:31:09 dist-test-slave-vpvm krb5kdc[25604](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332669, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.161.1@KRBTEST.COM for kudu/127.27.161.62@KRBTEST.COM
I20260810 03:31:09.355754 25980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:31:09.335994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.161.1:44721 (local address 127.27.161.62:43583)
0810 03:31:09.336490 (+ 496us) server_negotiation.cc:206] Beginning negotiation
0810 03:31:09.336497 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:31:09.337451 (+ 954us) server_negotiation.cc:407] Connection header received
0810 03:31:09.339452 (+ 2001us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:31:09.339459 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:31:09.339556 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:31:09.339658 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:31:09.341673 (+ 2015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:31:09.342724 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:31:09.344392 (+ 1668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:31:09.344966 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:31:09.345098 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:31:09.348965 (+ 3867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:31:09.348973 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:31:09.348978 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:31:09.349022 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:31:09.352497 (+ 3475us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:31:09.353348 (+ 851us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:31:09.353354 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:31:09.353357 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:31:09.353507 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:31:09.353937 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:31:09.353941 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:31:09.353944 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:31:09.354307 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:31:09.354451 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0810 03:31:09.355172 (+ 721us) server_negotiation.cc:299] Negotiation successful
0810 03:31:09.355401 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":360,"thread_start_us":217,"threads_started":1}
I20260810 03:31:09.356618 25981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:31:09.336533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:43583 (local address 127.27.161.1:44721)
0810 03:31:09.337293 (+ 760us) negotiation.cc:107] Waiting for socket to connect
0810 03:31:09.337331 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0810 03:31:09.339108 (+ 1777us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:31:09.339946 (+ 838us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:31:09.339959 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:31:09.340661 (+ 702us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:31:09.341445 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:31:09.341473 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:31:09.342906 (+ 1433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:31:09.342910 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:31:09.344226 (+ 1316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:31:09.344240 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:31:09.345220 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:31:09.345226 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:31:09.345448 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:31:09.346181 (+ 733us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:31:09.346218 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:31:09.348676 (+ 2458us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:31:09.352685 (+ 4009us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:31:09.352693 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:31:09.352713 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:31:09.353216 (+ 503us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:31:09.353616 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:31:09.353619 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:31:09.353621 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:31:09.353835 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:31:09.354418 (+ 583us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:31:09.354425 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:31:09.354806 (+ 381us) client_negotiation.cc:769] Sending connection context
0810 03:31:09.355050 (+ 244us) client_negotiation.cc:240] Negotiation successful
0810 03:31:09.355471 (+ 421us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":512,"thread_start_us":291,"threads_started":1}
I20260810 03:31:09.358337 25977 heartbeater.cc:347] Connected to a master server at 127.27.161.62:43583
I20260810 03:31:09.358640 25977 heartbeater.cc:464] Registering TS with master...
I20260810 03:31:09.359465 25977 heartbeater.cc:511] Master 127.27.161.62:43583 requested a full tablet report, sending...
I20260810 03:31:09.361711 25664 ts_manager.cc:194] Registered new tserver with Master: 249100e6cc06451080ba73097f4b75c4 (127.27.161.1:35683)
I20260810 03:31:09.363284 25664 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.161.1@KRBTEST.COM'} at 127.27.161.1:44721
WARNING: no policy specified for kudu/127.27.161.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.161.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.161.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.2.keytab.
Entry for principal kudu/127.27.161.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.2.keytab.
I20260810 03:31:09.416535 28292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iBH5/build/asan/bin/kudu
/tmp/dist-test-task24iBH5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786332336319544-28292-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786332336319544-28292-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786332336319544-28292-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786332336319544-28292-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.161.2:0
--local_ip_for_outbound_sockets=127.27.161.2
--webserver_interface=127.27.161.2
--webserver_port=0
--tserver_master_addrs=127.27.161.62:43583
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.2.keytab
--principal=kudu/127.27.161.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.161.20:40853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786332336319544-28292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786332336319544-28292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786332336319544-28292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:31:09.776670 25985 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:31:09.776978 25985 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:31:09.777076 25985 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:31:09.784571 25985 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:31:09.784701 25985 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:31:09.784817 25985 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.161.2
I20260810 03:31:09.795043 25985 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.161.20:40853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786332336319544-28292-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786332336319544-28292-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.2.keytab
--principal=kudu/127.27.161.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.161.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786332336319544-28292-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.161.2
--webserver_port=0
--tserver_master_addrs=127.27.161.62:43583
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.161.2
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786332336319544-28292-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:31:09.796361 25985 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:31:09.797691 25985 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:31:09.811352 25992 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:31:09.811281 25997 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:31:09.811319 25991 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:31:09.812507 25985 server_base.cc:1034] running on GCE node
I20260810 03:31:09.813148 25985 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:31:09.814170 25985 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:31:09.815418 25985 hybrid_clock.cc:648] HybridClock initialized: now 1786332669815342 us; error 52 us; skew 500 ppm
I20260810 03:31:09.815791 25985 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:31:09 dist-test-slave-vpvm krb5kdc[25604](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332669, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.161.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:31:09.819355 25985 init.cc:377] Logged in from keytab as kudu/127.27.161.2@KRBTEST.COM (short username kudu)
I20260810 03:31:09.820909 25985 webserver.cc:460] Webserver started at http://127.27.161.2:45615/ using document root <none> and password file <none>
I20260810 03:31:09.821658 25985 fs_manager.cc:359] Metadata directory not provided
I20260810 03:31:09.821768 25985 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:31:09.822124 25985 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:31:09.825687 25985 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786332336319544-28292-0/minicluster-data/ts-1/data/instance:
uuid: "b067709aaa7647cb8b9efd1b4d800649"
format_stamp: "Formatted at 2026-08-10 03:31:09 on dist-test-slave-vpvm"
server_key: "12033a16245ca27531acb08929e9def5"
server_key_iv: "044d2b5394d790ed271cc739f078046a"
server_key_version: "encryptionkey@0"
I20260810 03:31:09.826714 25985 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786332336319544-28292-0/minicluster-data/ts-1/wal/instance:
uuid: "b067709aaa7647cb8b9efd1b4d800649"
format_stamp: "Formatted at 2026-08-10 03:31:09 on dist-test-slave-vpvm"
server_key: "12033a16245ca27531acb08929e9def5"
server_key_iv: "044d2b5394d790ed271cc739f078046a"
server_key_version: "encryptionkey@0"
I20260810 03:31:09.833058 25985 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260810 03:31:09.837373 26005 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:31:09.838915 25985 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 03:31:09.839111 25985 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786332336319544-28292-0/minicluster-data/ts-1/data,/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786332336319544-28292-0/minicluster-data/ts-1/wal
uuid: "b067709aaa7647cb8b9efd1b4d800649"
format_stamp: "Formatted at 2026-08-10 03:31:09 on dist-test-slave-vpvm"
server_key: "12033a16245ca27531acb08929e9def5"
server_key_iv: "044d2b5394d790ed271cc739f078046a"
server_key_version: "encryptionkey@0"
I20260810 03:31:09.839346 25985 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786332336319544-28292-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786332336319544-28292-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786332336319544-28292-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:31:09.856381 25985 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:31:09.861153 25985 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:31:09.861480 25985 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:31:09.862638 25985 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:31:09.864199 25985 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:31:09.864282 25985 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:31:09.864398 25985 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:31:09.864450 25985 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:31:09.897344 25985 rpc_server.cc:307] RPC server started. Bound to: 127.27.161.2:41465
I20260810 03:31:09.897384 26171 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.161.2:41465 every 8 connection(s)
I20260810 03:31:09.898698 25985 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786332336319544-28292-0/minicluster-data/ts-1/data/info.pb
I20260810 03:31:09.903075 28292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iBH5/build/asan/bin/kudu as pid 25985
I20260810 03:31:09.903312 28292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786332336319544-28292-0/minicluster-data/ts-1/wal/instance
I20260810 03:31:09.903746 28292 external_mini_cluster.cc:1428] Setting key 3829103c0e76885f1b869aa303c3f4df
Aug 10 03:31:09 dist-test-slave-vpvm krb5kdc[25604](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332669, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.161.2@KRBTEST.COM for kudu/127.27.161.62@KRBTEST.COM
I20260810 03:31:09.920964 26176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:31:09.901282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.161.2:49963 (local address 127.27.161.62:43583)
0810 03:31:09.901790 (+ 508us) server_negotiation.cc:206] Beginning negotiation
0810 03:31:09.901796 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:31:09.902787 (+ 991us) server_negotiation.cc:407] Connection header received
0810 03:31:09.904883 (+ 2096us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:31:09.904887 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:31:09.904956 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:31:09.905050 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:31:09.906983 (+ 1933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:31:09.907994 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:31:09.909687 (+ 1693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:31:09.910320 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:31:09.910451 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:31:09.914219 (+ 3768us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:31:09.914226 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:31:09.914230 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:31:09.914275 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:31:09.917386 (+ 3111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:31:09.918360 (+ 974us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:31:09.918366 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:31:09.918370 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:31:09.918543 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:31:09.919020 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:31:09.919025 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:31:09.919028 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:31:09.919385 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:31:09.919534 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0810 03:31:09.920353 (+ 819us) server_negotiation.cc:299] Negotiation successful
0810 03:31:09.920590 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":364,"thread_start_us":236,"threads_started":1}
I20260810 03:31:09.921875 26177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:31:09.901784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:43583 (local address 127.27.161.2:49963)
0810 03:31:09.902616 (+ 832us) negotiation.cc:107] Waiting for socket to connect
0810 03:31:09.902664 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0810 03:31:09.904614 (+ 1950us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:31:09.905334 (+ 720us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:31:09.905346 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:31:09.906126 (+ 780us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:31:09.906787 (+ 661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:31:09.906808 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:31:09.908210 (+ 1402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:31:09.908215 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:31:09.909520 (+ 1305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:31:09.909536 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:31:09.910548 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:31:09.910554 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:31:09.910796 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:31:09.911514 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:31:09.911550 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:31:09.913887 (+ 2337us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:31:09.917591 (+ 3704us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:31:09.917600 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:31:09.917616 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:31:09.918207 (+ 591us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:31:09.918676 (+ 469us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:31:09.918679 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:31:09.918681 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:31:09.918900 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:31:09.919550 (+ 650us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:31:09.919558 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:31:09.919946 (+ 388us) client_negotiation.cc:769] Sending connection context
0810 03:31:09.920215 (+ 269us) client_negotiation.cc:240] Negotiation successful
0810 03:31:09.920662 (+ 447us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":567,"thread_start_us":363,"threads_started":1}
I20260810 03:31:09.923715 26172 heartbeater.cc:347] Connected to a master server at 127.27.161.62:43583
I20260810 03:31:09.924062 26172 heartbeater.cc:464] Registering TS with master...
I20260810 03:31:09.925019 26172 heartbeater.cc:511] Master 127.27.161.62:43583 requested a full tablet report, sending...
I20260810 03:31:09.927089 25664 ts_manager.cc:194] Registered new tserver with Master: b067709aaa7647cb8b9efd1b4d800649 (127.27.161.2:41465)
I20260810 03:31:09.928192 25664 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.161.2@KRBTEST.COM'} at 127.27.161.2:49963
WARNING: no policy specified for kudu/127.27.161.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.161.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.161.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.3.keytab.
Entry for principal kudu/127.27.161.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.3.keytab.
I20260810 03:31:09.985136 28292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iBH5/build/asan/bin/kudu
/tmp/dist-test-task24iBH5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786332336319544-28292-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786332336319544-28292-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786332336319544-28292-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786332336319544-28292-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.161.3:0
--local_ip_for_outbound_sockets=127.27.161.3
--webserver_interface=127.27.161.3
--webserver_port=0
--tserver_master_addrs=127.27.161.62:43583
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.3.keytab
--principal=kudu/127.27.161.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.161.20:40853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786332336319544-28292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786332336319544-28292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786332336319544-28292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:31:10.350416 26180 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:31:10.350805 26180 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:31:10.350981 26180 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:31:10.358531 26180 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:31:10.358644 26180 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:31:10.358795 26180 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.161.3
I20260810 03:31:10.367203 25977 heartbeater.cc:503] Master 127.27.161.62:43583 was elected leader, sending a full tablet report...
I20260810 03:31:10.369045 26180 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.161.20:40853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786332336319544-28292-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786332336319544-28292-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.3.keytab
--principal=kudu/127.27.161.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.161.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786332336319544-28292-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.161.3
--webserver_port=0
--tserver_master_addrs=127.27.161.62:43583
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.161.3
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786332336319544-28292-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:31:10.370441 26180 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:31:10.371745 26180 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:31:10.385548 26189 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:31:10.385492 26190 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:31:10.385578 26193 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:31:10.386543 26180 server_base.cc:1034] running on GCE node
I20260810 03:31:10.387254 26180 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:31:10.388346 26180 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:31:10.389557 26180 hybrid_clock.cc:648] HybridClock initialized: now 1786332670389507 us; error 46 us; skew 500 ppm
I20260810 03:31:10.389902 26180 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:31:10 dist-test-slave-vpvm krb5kdc[25604](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332670, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.161.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:31:10.393193 26180 init.cc:377] Logged in from keytab as kudu/127.27.161.3@KRBTEST.COM (short username kudu)
I20260810 03:31:10.394804 26180 webserver.cc:460] Webserver started at http://127.27.161.3:44751/ using document root <none> and password file <none>
I20260810 03:31:10.395495 26180 fs_manager.cc:359] Metadata directory not provided
I20260810 03:31:10.395612 26180 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:31:10.395911 26180 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:31:10.398975 26180 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786332336319544-28292-0/minicluster-data/ts-2/data/instance:
uuid: "f8c6771f4d2747a3830a0b5681974591"
format_stamp: "Formatted at 2026-08-10 03:31:10 on dist-test-slave-vpvm"
server_key: "5d668f2e9a6cde4b0e1b04c0384b1931"
server_key_iv: "be7f9ccae87e4a1cf130a8fe56920e12"
server_key_version: "encryptionkey@0"
I20260810 03:31:10.399724 26180 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786332336319544-28292-0/minicluster-data/ts-2/wal/instance:
uuid: "f8c6771f4d2747a3830a0b5681974591"
format_stamp: "Formatted at 2026-08-10 03:31:10 on dist-test-slave-vpvm"
server_key: "5d668f2e9a6cde4b0e1b04c0384b1931"
server_key_iv: "be7f9ccae87e4a1cf130a8fe56920e12"
server_key_version: "encryptionkey@0"
I20260810 03:31:10.405505 26180 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260810 03:31:10.409426 26207 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:31:10.410917 26180 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 03:31:10.411077 26180 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786332336319544-28292-0/minicluster-data/ts-2/data,/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786332336319544-28292-0/minicluster-data/ts-2/wal
uuid: "f8c6771f4d2747a3830a0b5681974591"
format_stamp: "Formatted at 2026-08-10 03:31:10 on dist-test-slave-vpvm"
server_key: "5d668f2e9a6cde4b0e1b04c0384b1931"
server_key_iv: "be7f9ccae87e4a1cf130a8fe56920e12"
server_key_version: "encryptionkey@0"
I20260810 03:31:10.411274 26180 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786332336319544-28292-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786332336319544-28292-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786332336319544-28292-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:31:10.434669 26180 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:31:10.438902 26180 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:31:10.439190 26180 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:31:10.440230 26180 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:31:10.441587 26180 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:31:10.441666 26180 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:31:10.441778 26180 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:31:10.441825 26180 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:31:10.472782 26180 rpc_server.cc:307] RPC server started. Bound to: 127.27.161.3:33741
I20260810 03:31:10.472832 26387 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.161.3:33741 every 8 connection(s)
I20260810 03:31:10.474148 26180 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786332336319544-28292-0/minicluster-data/ts-2/data/info.pb
I20260810 03:31:10.481937 28292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iBH5/build/asan/bin/kudu as pid 26180
I20260810 03:31:10.482182 28292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786332336319544-28292-0/minicluster-data/ts-2/wal/instance
I20260810 03:31:10.482697 28292 external_mini_cluster.cc:1428] Setting key 774ca504b046f46124312eea1261331b
Aug 10 03:31:10 dist-test-slave-vpvm krb5kdc[25604](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332670, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.161.3@KRBTEST.COM for kudu/127.27.161.62@KRBTEST.COM
I20260810 03:31:10.495699 26392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:31:10.476728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.161.3:58481 (local address 127.27.161.62:43583)
0810 03:31:10.477160 (+ 432us) server_negotiation.cc:206] Beginning negotiation
0810 03:31:10.477164 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:31:10.478122 (+ 958us) server_negotiation.cc:407] Connection header received
0810 03:31:10.480012 (+ 1890us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:31:10.480018 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:31:10.480112 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:31:10.480214 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:31:10.481979 (+ 1765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:31:10.483318 (+ 1339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:31:10.485322 (+ 2004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:31:10.485740 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:31:10.485866 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:31:10.489734 (+ 3868us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:31:10.489739 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:31:10.489743 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:31:10.489790 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:31:10.492440 (+ 2650us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:31:10.493319 (+ 879us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:31:10.493324 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:31:10.493327 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:31:10.493472 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:31:10.494021 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:31:10.494027 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:31:10.494031 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:31:10.494329 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:31:10.494455 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0810 03:31:10.495212 (+ 757us) server_negotiation.cc:299] Negotiation successful
0810 03:31:10.495406 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":211,"threads_started":1}
I20260810 03:31:10.496663 26394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:31:10.477183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:43583 (local address 127.27.161.3:58481)
0810 03:31:10.477914 (+ 731us) negotiation.cc:107] Waiting for socket to connect
0810 03:31:10.477991 (+ 77us) client_negotiation.cc:174] Beginning negotiation
0810 03:31:10.479726 (+ 1735us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:31:10.480473 (+ 747us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:31:10.480482 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:31:10.481101 (+ 619us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:31:10.481768 (+ 667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:31:10.481789 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:31:10.483516 (+ 1727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:31:10.483523 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:31:10.485142 (+ 1619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:31:10.485157 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:31:10.486020 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:31:10.486031 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:31:10.486273 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:31:10.487020 (+ 747us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:31:10.487048 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:31:10.489455 (+ 2407us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:31:10.492618 (+ 3163us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:31:10.492627 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:31:10.492643 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:31:10.493181 (+ 538us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:31:10.493628 (+ 447us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:31:10.493633 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:31:10.493636 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:31:10.493860 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:31:10.494444 (+ 584us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:31:10.494450 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:31:10.494824 (+ 374us) client_negotiation.cc:769] Sending connection context
0810 03:31:10.495106 (+ 282us) client_negotiation.cc:240] Negotiation successful
0810 03:31:10.495542 (+ 436us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":492,"thread_start_us":273,"threads_started":1}
I20260810 03:31:10.498203 26388 heartbeater.cc:347] Connected to a master server at 127.27.161.62:43583
I20260810 03:31:10.498592 26388 heartbeater.cc:464] Registering TS with master...
I20260810 03:31:10.499441 26388 heartbeater.cc:511] Master 127.27.161.62:43583 requested a full tablet report, sending...
I20260810 03:31:10.500950 25663 ts_manager.cc:194] Registered new tserver with Master: f8c6771f4d2747a3830a0b5681974591 (127.27.161.3:33741)
I20260810 03:31:10.501854 25663 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.161.3@KRBTEST.COM'} at 127.27.161.3:58481
I20260810 03:31:10.511924 28292 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 03:31:10.529549 26392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:31:10.516521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47198 (local address 127.27.161.62:43583)
0810 03:31:10.516746 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0810 03:31:10.516754 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 03:31:10.517111 (+ 357us) server_negotiation.cc:407] Connection header received
0810 03:31:10.517277 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:31:10.517281 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:31:10.517354 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:31:10.517440 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:31:10.518991 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:31:10.519880 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:31:10.521446 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:31:10.521884 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:31:10.522055 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:31:10.524515 (+ 2460us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:31:10.524520 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:31:10.524523 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:31:10.524560 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:31:10.527019 (+ 2459us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:31:10.527785 (+ 766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:31:10.527789 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:31:10.527791 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:31:10.527909 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:31:10.528252 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:31:10.528254 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:31:10.528256 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:31:10.528578 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:31:10.528773 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0810 03:31:10.529103 (+ 330us) server_negotiation.cc:299] Negotiation successful
0810 03:31:10.529282 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260810 03:31:10.533501 25663 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47198:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 03:31:10.540361 25663 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 03:31:10.564844 26413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:31:10.553090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.2:41465 (local address 127.0.0.1:48334)
0810 03:31:10.554674 (+ 1584us) negotiation.cc:107] Waiting for socket to connect
0810 03:31:10.554711 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0810 03:31:10.554887 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:31:10.556721 (+ 1834us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:31:10.556727 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:31:10.556756 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:31:10.557479 (+ 723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:31:10.557493 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:31:10.559431 (+ 1938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:31:10.559436 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:31:10.562039 (+ 2603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:31:10.562055 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:31:10.563850 (+ 1795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:31:10.563856 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:31:10.564022 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:31:10.564060 (+ 38us) client_negotiation.cc:769] Sending connection context
0810 03:31:10.564140 (+ 80us) client_negotiation.cc:240] Negotiation successful
0810 03:31:10.564399 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1314,"mutex_wait_us":136,"thread_start_us":1157,"threads_started":1}
I20260810 03:31:10.564918 26418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:31:10.554301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41032 (local address 127.27.161.3:33741)
0810 03:31:10.555870 (+ 1569us) server_negotiation.cc:206] Beginning negotiation
0810 03:31:10.555878 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 03:31:10.555915 (+ 37us) server_negotiation.cc:407] Connection header received
0810 03:31:10.556016 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:31:10.556024 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:31:10.556320 (+ 296us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:31:10.556709 (+ 389us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:31:10.558209 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:31:10.559658 (+ 1449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:31:10.562091 (+ 2433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:31:10.563526 (+ 1435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:31:10.563665 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:31:10.563923 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0810 03:31:10.564462 (+ 539us) server_negotiation.cc:299] Negotiation successful
0810 03:31:10.564635 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1400,"thread_start_us":1259,"threads_started":1}
I20260810 03:31:10.565068 26415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:31:10.553279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48334 (local address 127.27.161.2:41465)
0810 03:31:10.556103 (+ 2824us) server_negotiation.cc:206] Beginning negotiation
0810 03:31:10.556113 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 03:31:10.556143 (+ 30us) server_negotiation.cc:407] Connection header received
0810 03:31:10.556253 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:31:10.556260 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:31:10.556478 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:31:10.557191 (+ 713us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:31:10.557669 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:31:10.559291 (+ 1622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:31:10.562218 (+ 2927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:31:10.563610 (+ 1392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:31:10.564297 (+ 687us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:31:10.564479 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0810 03:31:10.564578 (+ 99us) server_negotiation.cc:299] Negotiation successful
0810 03:31:10.564746 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2666,"thread_start_us":2538,"threads_started":1}
I20260810 03:31:10.565618 26417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:31:10.553551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.3:33741 (local address 127.0.0.1:41032)
0810 03:31:10.555471 (+ 1920us) negotiation.cc:107] Waiting for socket to connect
0810 03:31:10.555488 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 03:31:10.555613 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:31:10.556565 (+ 952us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:31:10.556572 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:31:10.556604 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:31:10.558004 (+ 1400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:31:10.558017 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:31:10.559789 (+ 1772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:31:10.559795 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:31:10.561891 (+ 2096us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:31:10.561904 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:31:10.563737 (+ 1833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:31:10.563743 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:31:10.563964 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:31:10.564028 (+ 64us) client_negotiation.cc:769] Sending connection context
0810 03:31:10.564167 (+ 139us) client_negotiation.cc:240] Negotiation successful
0810 03:31:10.565213 (+ 1046us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1794,"mutex_wait_us":882,"thread_start_us":1623,"threads_started":1}
I20260810 03:31:10.566045 26419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:31:10.555673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.1:35683 (local address 127.0.0.1:58088)
0810 03:31:10.557084 (+ 1411us) negotiation.cc:107] Waiting for socket to connect
0810 03:31:10.557101 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 03:31:10.557207 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:31:10.558697 (+ 1490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:31:10.558701 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:31:10.558725 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:31:10.559038 (+ 313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:31:10.559047 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:31:10.560833 (+ 1786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:31:10.560839 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:31:10.562689 (+ 1850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:31:10.562701 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:31:10.564089 (+ 1388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:31:10.564096 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:31:10.564254 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:31:10.564282 (+ 28us) client_negotiation.cc:769] Sending connection context
0810 03:31:10.565473 (+ 1191us) client_negotiation.cc:240] Negotiation successful
0810 03:31:10.565622 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1279,"mutex_wait_us":32,"thread_start_us":1109,"threads_started":1}
I20260810 03:31:10.566589 26421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:31:10.556306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58088 (local address 127.27.161.1:35683)
0810 03:31:10.558244 (+ 1938us) server_negotiation.cc:206] Beginning negotiation
0810 03:31:10.558252 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 03:31:10.558274 (+ 22us) server_negotiation.cc:407] Connection header received
0810 03:31:10.558351 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:31:10.558356 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:31:10.558522 (+ 166us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:31:10.558682 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:31:10.559190 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:31:10.560653 (+ 1463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:31:10.562836 (+ 2183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:31:10.563847 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:31:10.563973 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:31:10.564190 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0810 03:31:10.566065 (+ 1875us) server_negotiation.cc:299] Negotiation successful
0810 03:31:10.566251 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1739,"thread_start_us":1536,"threads_started":1}
I20260810 03:31:10.569087 26077 tablet_service.cc:1467] Processing CreateTablet for tablet f611075fd74643dc84bf59471b18a55c (DEFAULT_TABLE table=test-table [id=758d9b69e34040d5a4d32be5ca3ab8ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:31:10.569326 26289 tablet_service.cc:1467] Processing CreateTablet for tablet f611075fd74643dc84bf59471b18a55c (DEFAULT_TABLE table=test-table [id=758d9b69e34040d5a4d32be5ca3ab8ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:31:10.570590 26077 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f611075fd74643dc84bf59471b18a55c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:31:10.570170 25876 tablet_service.cc:1467] Processing CreateTablet for tablet f611075fd74643dc84bf59471b18a55c (DEFAULT_TABLE table=test-table [id=758d9b69e34040d5a4d32be5ca3ab8ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:31:10.570760 26289 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f611075fd74643dc84bf59471b18a55c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:31:10.571489 25876 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f611075fd74643dc84bf59471b18a55c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:31:10.582404 26422 tablet_bootstrap.cc:492] T f611075fd74643dc84bf59471b18a55c P b067709aaa7647cb8b9efd1b4d800649: Bootstrap starting.
I20260810 03:31:10.586557 26423 tablet_bootstrap.cc:492] T f611075fd74643dc84bf59471b18a55c P f8c6771f4d2747a3830a0b5681974591: Bootstrap starting.
I20260810 03:31:10.587657 26422 tablet_bootstrap.cc:654] T f611075fd74643dc84bf59471b18a55c P b067709aaa7647cb8b9efd1b4d800649: Neither blocks nor log segments found. Creating new log.
I20260810 03:31:10.589171 26422 log.cc:824] T f611075fd74643dc84bf59471b18a55c P b067709aaa7647cb8b9efd1b4d800649: Log is configured to *not* fsync() on all Append() calls
I20260810 03:31:10.590425 26423 tablet_bootstrap.cc:654] T f611075fd74643dc84bf59471b18a55c P f8c6771f4d2747a3830a0b5681974591: Neither blocks nor log segments found. Creating new log.
I20260810 03:31:10.591450 26426 tablet_bootstrap.cc:492] T f611075fd74643dc84bf59471b18a55c P 249100e6cc06451080ba73097f4b75c4: Bootstrap starting.
I20260810 03:31:10.591894 26423 log.cc:824] T f611075fd74643dc84bf59471b18a55c P f8c6771f4d2747a3830a0b5681974591: Log is configured to *not* fsync() on all Append() calls
I20260810 03:31:10.593693 26422 tablet_bootstrap.cc:492] T f611075fd74643dc84bf59471b18a55c P b067709aaa7647cb8b9efd1b4d800649: No bootstrap required, opened a new log
I20260810 03:31:10.594050 26422 ts_tablet_manager.cc:1397] T f611075fd74643dc84bf59471b18a55c P b067709aaa7647cb8b9efd1b4d800649: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.003s
I20260810 03:31:10.594867 26426 tablet_bootstrap.cc:654] T f611075fd74643dc84bf59471b18a55c P 249100e6cc06451080ba73097f4b75c4: Neither blocks nor log segments found. Creating new log.
I20260810 03:31:10.595906 26426 log.cc:824] T f611075fd74643dc84bf59471b18a55c P 249100e6cc06451080ba73097f4b75c4: Log is configured to *not* fsync() on all Append() calls
I20260810 03:31:10.596792 26423 tablet_bootstrap.cc:492] T f611075fd74643dc84bf59471b18a55c P f8c6771f4d2747a3830a0b5681974591: No bootstrap required, opened a new log
I20260810 03:31:10.597090 26423 ts_tablet_manager.cc:1397] T f611075fd74643dc84bf59471b18a55c P f8c6771f4d2747a3830a0b5681974591: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.003s
I20260810 03:31:10.598544 26426 tablet_bootstrap.cc:492] T f611075fd74643dc84bf59471b18a55c P 249100e6cc06451080ba73097f4b75c4: No bootstrap required, opened a new log
I20260810 03:31:10.598798 26426 ts_tablet_manager.cc:1397] T f611075fd74643dc84bf59471b18a55c P 249100e6cc06451080ba73097f4b75c4: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260810 03:31:10.605464 26422 raft_consensus.cc:348] T f611075fd74643dc84bf59471b18a55c P b067709aaa7647cb8b9efd1b4d800649 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b067709aaa7647cb8b9efd1b4d800649" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 41465 } } peers { permanent_uuid: "f8c6771f4d2747a3830a0b5681974591" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 33741 } } peers { permanent_uuid: "249100e6cc06451080ba73097f4b75c4" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 35683 } }
I20260810 03:31:10.606151 26422 raft_consensus.cc:374] T f611075fd74643dc84bf59471b18a55c P b067709aaa7647cb8b9efd1b4d800649 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:31:10.606285 26422 raft_consensus.cc:729] T f611075fd74643dc84bf59471b18a55c P b067709aaa7647cb8b9efd1b4d800649 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b067709aaa7647cb8b9efd1b4d800649, State: Initialized, Role: FOLLOWER
I20260810 03:31:10.607154 26422 consensus_queue.cc:260] T f611075fd74643dc84bf59471b18a55c P b067709aaa7647cb8b9efd1b4d800649 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b067709aaa7647cb8b9efd1b4d800649" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 41465 } } peers { permanent_uuid: "f8c6771f4d2747a3830a0b5681974591" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 33741 } } peers { permanent_uuid: "249100e6cc06451080ba73097f4b75c4" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 35683 } }
I20260810 03:31:10.608055 26423 raft_consensus.cc:348] T f611075fd74643dc84bf59471b18a55c P f8c6771f4d2747a3830a0b5681974591 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b067709aaa7647cb8b9efd1b4d800649" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 41465 } } peers { permanent_uuid: "f8c6771f4d2747a3830a0b5681974591" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 33741 } } peers { permanent_uuid: "249100e6cc06451080ba73097f4b75c4" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 35683 } }
I20260810 03:31:10.608567 26423 raft_consensus.cc:374] T f611075fd74643dc84bf59471b18a55c P f8c6771f4d2747a3830a0b5681974591 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:31:10.608681 26423 raft_consensus.cc:729] T f611075fd74643dc84bf59471b18a55c P f8c6771f4d2747a3830a0b5681974591 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8c6771f4d2747a3830a0b5681974591, State: Initialized, Role: FOLLOWER
I20260810 03:31:10.608703 26172 heartbeater.cc:503] Master 127.27.161.62:43583 was elected leader, sending a full tablet report...
I20260810 03:31:10.608623 26426 raft_consensus.cc:348] T f611075fd74643dc84bf59471b18a55c P 249100e6cc06451080ba73097f4b75c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b067709aaa7647cb8b9efd1b4d800649" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 41465 } } peers { permanent_uuid: "f8c6771f4d2747a3830a0b5681974591" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 33741 } } peers { permanent_uuid: "249100e6cc06451080ba73097f4b75c4" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 35683 } }
I20260810 03:31:10.609015 26426 raft_consensus.cc:374] T f611075fd74643dc84bf59471b18a55c P 249100e6cc06451080ba73097f4b75c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:31:10.609102 26426 raft_consensus.cc:729] T f611075fd74643dc84bf59471b18a55c P 249100e6cc06451080ba73097f4b75c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 249100e6cc06451080ba73097f4b75c4, State: Initialized, Role: FOLLOWER
I20260810 03:31:10.609459 26423 consensus_queue.cc:260] T f611075fd74643dc84bf59471b18a55c P f8c6771f4d2747a3830a0b5681974591 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b067709aaa7647cb8b9efd1b4d800649" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 41465 } } peers { permanent_uuid: "f8c6771f4d2747a3830a0b5681974591" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 33741 } } peers { permanent_uuid: "249100e6cc06451080ba73097f4b75c4" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 35683 } }
I20260810 03:31:10.609644 26426 consensus_queue.cc:260] T f611075fd74643dc84bf59471b18a55c P 249100e6cc06451080ba73097f4b75c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b067709aaa7647cb8b9efd1b4d800649" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 41465 } } peers { permanent_uuid: "f8c6771f4d2747a3830a0b5681974591" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 33741 } } peers { permanent_uuid: "249100e6cc06451080ba73097f4b75c4" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 35683 } }
I20260810 03:31:10.610628 26388 heartbeater.cc:503] Master 127.27.161.62:43583 was elected leader, sending a full tablet report...
I20260810 03:31:10.611565 26422 ts_tablet_manager.cc:1428] T f611075fd74643dc84bf59471b18a55c P b067709aaa7647cb8b9efd1b4d800649: Time spent starting tablet: real 0.017s user 0.010s sys 0.003s
I20260810 03:31:10.613240 26426 ts_tablet_manager.cc:1428] T f611075fd74643dc84bf59471b18a55c P 249100e6cc06451080ba73097f4b75c4: Time spent starting tablet: real 0.014s user 0.015s sys 0.000s
I20260810 03:31:10.613457 26423 ts_tablet_manager.cc:1428] T f611075fd74643dc84bf59471b18a55c P f8c6771f4d2747a3830a0b5681974591: Time spent starting tablet: real 0.016s user 0.015s sys 0.001s
W20260810 03:31:10.650707 26174 tablet.cc:2369] T f611075fd74643dc84bf59471b18a55c P b067709aaa7647cb8b9efd1b4d800649: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 03:31:10.725845 26390 tablet.cc:2369] T f611075fd74643dc84bf59471b18a55c P f8c6771f4d2747a3830a0b5681974591: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 03:31:10.836216 25979 tablet.cc:2369] T f611075fd74643dc84bf59471b18a55c P 249100e6cc06451080ba73097f4b75c4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 03:31:10.844121 26431 raft_consensus.cc:482] T f611075fd74643dc84bf59471b18a55c P b067709aaa7647cb8b9efd1b4d800649 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 03:31:10.844339 26432 raft_consensus.cc:482] T f611075fd74643dc84bf59471b18a55c P f8c6771f4d2747a3830a0b5681974591 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 03:31:10.844463 26431 raft_consensus.cc:504] T f611075fd74643dc84bf59471b18a55c P b067709aaa7647cb8b9efd1b4d800649 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b067709aaa7647cb8b9efd1b4d800649" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 41465 } } peers { permanent_uuid: "f8c6771f4d2747a3830a0b5681974591" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 33741 } } peers { permanent_uuid: "249100e6cc06451080ba73097f4b75c4" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 35683 } }
I20260810 03:31:10.844739 26432 raft_consensus.cc:504] T f611075fd74643dc84bf59471b18a55c P f8c6771f4d2747a3830a0b5681974591 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b067709aaa7647cb8b9efd1b4d800649" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 41465 } } peers { permanent_uuid: "f8c6771f4d2747a3830a0b5681974591" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 33741 } } peers { permanent_uuid: "249100e6cc06451080ba73097f4b75c4" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 35683 } }
I20260810 03:31:10.846529 26431 leader_election.cc:288] T f611075fd74643dc84bf59471b18a55c P b067709aaa7647cb8b9efd1b4d800649 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f8c6771f4d2747a3830a0b5681974591 (127.27.161.3:33741), 249100e6cc06451080ba73097f4b75c4 (127.27.161.1:35683)
I20260810 03:31:10.846534 26432 leader_election.cc:288] T f611075fd74643dc84bf59471b18a55c P f8c6771f4d2747a3830a0b5681974591 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b067709aaa7647cb8b9efd1b4d800649 (127.27.161.2:41465), 249100e6cc06451080ba73097f4b75c4 (127.27.161.1:35683)
I20260810 03:31:10.854415 26394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:31:10.846867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.2:41465 (local address 127.27.161.3:45633)
0810 03:31:10.847226 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0810 03:31:10.847251 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0810 03:31:10.847393 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:31:10.847914 (+ 521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:31:10.847919 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:31:10.848000 (+ 81us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:31:10.848309 (+ 309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:31:10.848321 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:31:10.849711 (+ 1390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:31:10.849717 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:31:10.851628 (+ 1911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:31:10.851642 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:31:10.853515 (+ 1873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:31:10.853521 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:31:10.853735 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:31:10.853768 (+ 33us) client_negotiation.cc:769] Sending connection context
0810 03:31:10.853865 (+ 97us) client_negotiation.cc:240] Negotiation successful
0810 03:31:10.854050 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":218,"mutex_wait_us":39,"spinlock_wait_cycles":27392}
I20260810 03:31:10.855182 26415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:31:10.846928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.161.3:45633 (local address 127.27.161.2:41465)
0810 03:31:10.847117 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0810 03:31:10.847123 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:31:10.847515 (+ 392us) server_negotiation.cc:407] Connection header received
0810 03:31:10.847595 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:31:10.847601 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:31:10.847680 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:31:10.847758 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:31:10.848473 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:31:10.849538 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:31:10.851868 (+ 2330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:31:10.853196 (+ 1328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:31:10.853339 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:31:10.853414 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0810 03:31:10.854756 (+ 1342us) server_negotiation.cc:299] Negotiation successful
0810 03:31:10.854903 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260810 03:31:10.856468 26105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f611075fd74643dc84bf59471b18a55c" candidate_uuid: "f8c6771f4d2747a3830a0b5681974591" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b067709aaa7647cb8b9efd1b4d800649" is_pre_election: true
I20260810 03:31:10.856982 26105 raft_consensus.cc:2454] T f611075fd74643dc84bf59471b18a55c P b067709aaa7647cb8b9efd1b4d800649 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8c6771f4d2747a3830a0b5681974591 in term 0.
I20260810 03:31:10.857821 26214 leader_election.cc:302] T f611075fd74643dc84bf59471b18a55c P f8c6771f4d2747a3830a0b5681974591 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b067709aaa7647cb8b9efd1b4d800649, f8c6771f4d2747a3830a0b5681974591; no voters:
I20260810 03:31:10.858004 26421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:31:10.847561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.161.3:43835 (local address 127.27.161.1:35683)
0810 03:31:10.847765 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0810 03:31:10.847771 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:31:10.850772 (+ 3001us) server_negotiation.cc:407] Connection header received
0810 03:31:10.850948 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:31:10.850954 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:31:10.851029 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:31:10.851140 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:31:10.851990 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:31:10.853103 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:31:10.855568 (+ 2465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:31:10.856742 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:31:10.856877 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:31:10.856946 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0810 03:31:10.857608 (+ 662us) server_negotiation.cc:299] Negotiation successful
0810 03:31:10.857748 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260810 03:31:10.858512 26432 raft_consensus.cc:2781] T f611075fd74643dc84bf59471b18a55c P f8c6771f4d2747a3830a0b5681974591 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 03:31:10.858762 26432 raft_consensus.cc:482] T f611075fd74643dc84bf59471b18a55c P f8c6771f4d2747a3830a0b5681974591 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 03:31:10.858871 26436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:31:10.847867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.1:35683 (local address 127.27.161.2:42329)
0810 03:31:10.850348 (+ 2481us) negotiation.cc:107] Waiting for socket to connect
0810 03:31:10.850366 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 03:31:10.850546 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:31:10.851757 (+ 1211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:31:10.851761 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:31:10.851791 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:31:10.852299 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:31:10.852311 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:31:10.853929 (+ 1618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:31:10.853934 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:31:10.856207 (+ 2273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:31:10.856223 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:31:10.858094 (+ 1871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:31:10.858101 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:31:10.858265 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:31:10.858306 (+ 41us) client_negotiation.cc:769] Sending connection context
0810 03:31:10.858390 (+ 84us) client_negotiation.cc:240] Negotiation successful
0810 03:31:10.858516 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2363,"mutex_wait_us":62,"thread_start_us":2140,"threads_started":1}
I20260810 03:31:10.858932 26432 raft_consensus.cc:3037] T f611075fd74643dc84bf59471b18a55c P f8c6771f4d2747a3830a0b5681974591 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:31:10.859030 26418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:31:10.847277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.161.2:55939 (local address 127.27.161.3:33741)
0810 03:31:10.847461 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0810 03:31:10.847466 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:31:10.847687 (+ 221us) server_negotiation.cc:407] Connection header received
0810 03:31:10.847902 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:31:10.847907 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:31:10.847974 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:31:10.848029 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:31:10.852832 (+ 4803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:31:10.853916 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:31:10.856463 (+ 2547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:31:10.857846 (+ 1383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:31:10.858010 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:31:10.858077 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0810 03:31:10.858681 (+ 604us) server_negotiation.cc:299] Negotiation successful
0810 03:31:10.858802 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":193,"server-negotiator.queue_time_us":57}
I20260810 03:31:10.859021 26437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:31:10.847922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.161.2:42329 (local address 127.27.161.1:35683)
0810 03:31:10.851443 (+ 3521us) server_negotiation.cc:206] Beginning negotiation
0810 03:31:10.851448 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:31:10.851469 (+ 21us) server_negotiation.cc:407] Connection header received
0810 03:31:10.851527 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:31:10.851531 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:31:10.851610 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:31:10.851710 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:31:10.852545 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:31:10.853777 (+ 1232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:31:10.856401 (+ 2624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:31:10.857661 (+ 1260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:31:10.857828 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:31:10.857896 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0810 03:31:10.858588 (+ 692us) server_negotiation.cc:299] Negotiation successful
0810 03:31:10.858717 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3394,"thread_start_us":3145,"threads_started":1}
I20260810 03:31:10.859838 26434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:31:10.846732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.3:33741 (local address 127.27.161.2:55939)
0810 03:31:10.847596 (+ 864us) negotiation.cc:107] Waiting for socket to connect
0810 03:31:10.847617 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0810 03:31:10.847767 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:31:10.852016 (+ 4249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:31:10.852020 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:31:10.852056 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:31:10.852651 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:31:10.852664 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:31:10.854135 (+ 1471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:31:10.854140 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:31:10.856288 (+ 2148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:31:10.856304 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:31:10.858296 (+ 1992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:31:10.858302 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:31:10.858481 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:31:10.858516 (+ 35us) client_negotiation.cc:769] Sending connection context
0810 03:31:10.859357 (+ 841us) client_negotiation.cc:240] Negotiation successful
0810 03:31:10.859494 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":723,"mutex_wait_us":71,"thread_start_us":739,"threads_started":1}
I20260810 03:31:10.859993 26435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:31:10.846980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.1:35683 (local address 127.27.161.3:43835)
0810 03:31:10.850645 (+ 3665us) negotiation.cc:107] Waiting for socket to connect
0810 03:31:10.850664 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 03:31:10.850811 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:31:10.851289 (+ 478us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:31:10.851293 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:31:10.851320 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:31:10.851768 (+ 448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:31:10.851783 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:31:10.853298 (+ 1515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:31:10.853303 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:31:10.855393 (+ 2090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:31:10.855408 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:31:10.857132 (+ 1724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:31:10.857139 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:31:10.857427 (+ 288us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:31:10.857460 (+ 33us) client_negotiation.cc:769] Sending connection context
0810 03:31:10.858231 (+ 771us) client_negotiation.cc:240] Negotiation successful
0810 03:31:10.858367 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3525,"mutex_wait_us":73,"thread_start_us":3362,"threads_started":1}
I20260810 03:31:10.860198 25905 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f611075fd74643dc84bf59471b18a55c" candidate_uuid: "b067709aaa7647cb8b9efd1b4d800649" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "249100e6cc06451080ba73097f4b75c4" is_pre_election: true
I20260810 03:31:10.860754 25905 raft_consensus.cc:2454] T f611075fd74643dc84bf59471b18a55c P 249100e6cc06451080ba73097f4b75c4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b067709aaa7647cb8b9efd1b4d800649 in term 0.
I20260810 03:31:10.860870 26317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f611075fd74643dc84bf59471b18a55c" candidate_uuid: "b067709aaa7647cb8b9efd1b4d800649" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8c6771f4d2747a3830a0b5681974591" is_pre_election: true
I20260810 03:31:10.860947 25904 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f611075fd74643dc84bf59471b18a55c" candidate_uuid: "f8c6771f4d2747a3830a0b5681974591" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "249100e6cc06451080ba73097f4b75c4" is_pre_election: true
I20260810 03:31:10.861266 25904 raft_consensus.cc:2454] T f611075fd74643dc84bf59471b18a55c P 249100e6cc06451080ba73097f4b75c4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8c6771f4d2747a3830a0b5681974591 in term 0.
I20260810 03:31:10.861496 26008 leader_election.cc:302] T f611075fd74643dc84bf59471b18a55c P b067709aaa7647cb8b9efd1b4d800649 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 249100e6cc06451080ba73097f4b75c4, b067709aaa7647cb8b9efd1b4d800649; no voters:
I20260810 03:31:10.861915 26431 raft_consensus.cc:2781] T f611075fd74643dc84bf59471b18a55c P b067709aaa7647cb8b9efd1b4d800649 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 03:31:10.862106 26431 raft_consensus.cc:482] T f611075fd74643dc84bf59471b18a55c P b067709aaa7647cb8b9efd1b4d800649 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 03:31:10.862229 26431 raft_consensus.cc:3037] T f611075fd74643dc84bf59471b18a55c P b067709aaa7647cb8b9efd1b4d800649 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:31:10.862229 26432 raft_consensus.cc:504] T f611075fd74643dc84bf59471b18a55c P f8c6771f4d2747a3830a0b5681974591 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b067709aaa7647cb8b9efd1b4d800649" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 41465 } } peers { permanent_uuid: "f8c6771f4d2747a3830a0b5681974591" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 33741 } } peers { permanent_uuid: "249100e6cc06451080ba73097f4b75c4" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 35683 } }
I20260810 03:31:10.862645 26317 raft_consensus.cc:2379] T f611075fd74643dc84bf59471b18a55c P f8c6771f4d2747a3830a0b5681974591 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b067709aaa7647cb8b9efd1b4d800649 in current term 1: Already voted for candidate f8c6771f4d2747a3830a0b5681974591 in this term.
I20260810 03:31:10.863286 25904 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f611075fd74643dc84bf59471b18a55c" candidate_uuid: "f8c6771f4d2747a3830a0b5681974591" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "249100e6cc06451080ba73097f4b75c4"
I20260810 03:31:10.863451 25904 raft_consensus.cc:3037] T f611075fd74643dc84bf59471b18a55c P 249100e6cc06451080ba73097f4b75c4 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:31:10.864960 26431 raft_consensus.cc:504] T f611075fd74643dc84bf59471b18a55c P b067709aaa7647cb8b9efd1b4d800649 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b067709aaa7647cb8b9efd1b4d800649" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 41465 } } peers { permanent_uuid: "f8c6771f4d2747a3830a0b5681974591" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 33741 } } peers { permanent_uuid: "249100e6cc06451080ba73097f4b75c4" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 35683 } }
I20260810 03:31:10.865442 26105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f611075fd74643dc84bf59471b18a55c" candidate_uuid: "f8c6771f4d2747a3830a0b5681974591" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b067709aaa7647cb8b9efd1b4d800649"
I20260810 03:31:10.865671 26431 leader_election.cc:288] T f611075fd74643dc84bf59471b18a55c P b067709aaa7647cb8b9efd1b4d800649 [CANDIDATE]: Term 1 election: Requested vote from peers f8c6771f4d2747a3830a0b5681974591 (127.27.161.3:33741), 249100e6cc06451080ba73097f4b75c4 (127.27.161.1:35683)
I20260810 03:31:10.865733 26105 raft_consensus.cc:2379] T f611075fd74643dc84bf59471b18a55c P b067709aaa7647cb8b9efd1b4d800649 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate f8c6771f4d2747a3830a0b5681974591 in current term 1: Already voted for candidate b067709aaa7647cb8b9efd1b4d800649 in this term.
I20260810 03:31:10.865862 26432 leader_election.cc:288] T f611075fd74643dc84bf59471b18a55c P f8c6771f4d2747a3830a0b5681974591 [CANDIDATE]: Term 1 election: Requested vote from peers b067709aaa7647cb8b9efd1b4d800649 (127.27.161.2:41465), 249100e6cc06451080ba73097f4b75c4 (127.27.161.1:35683)
I20260810 03:31:10.866182 25904 raft_consensus.cc:2454] T f611075fd74643dc84bf59471b18a55c P 249100e6cc06451080ba73097f4b75c4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8c6771f4d2747a3830a0b5681974591 in term 1.
I20260810 03:31:10.866303 26317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f611075fd74643dc84bf59471b18a55c" candidate_uuid: "b067709aaa7647cb8b9efd1b4d800649" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8c6771f4d2747a3830a0b5681974591"
I20260810 03:31:10.866549 26317 raft_consensus.cc:2379] T f611075fd74643dc84bf59471b18a55c P f8c6771f4d2747a3830a0b5681974591 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate b067709aaa7647cb8b9efd1b4d800649 in current term 1: Already voted for candidate f8c6771f4d2747a3830a0b5681974591 in this term.
I20260810 03:31:10.866608 25905 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f611075fd74643dc84bf59471b18a55c" candidate_uuid: "b067709aaa7647cb8b9efd1b4d800649" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "249100e6cc06451080ba73097f4b75c4"
I20260810 03:31:10.866648 26210 leader_election.cc:302] T f611075fd74643dc84bf59471b18a55c P f8c6771f4d2747a3830a0b5681974591 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 249100e6cc06451080ba73097f4b75c4, f8c6771f4d2747a3830a0b5681974591; no voters: b067709aaa7647cb8b9efd1b4d800649
I20260810 03:31:10.866863 25905 raft_consensus.cc:2379] T f611075fd74643dc84bf59471b18a55c P 249100e6cc06451080ba73097f4b75c4 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate b067709aaa7647cb8b9efd1b4d800649 in current term 1: Already voted for candidate f8c6771f4d2747a3830a0b5681974591 in this term.
I20260810 03:31:10.867066 26432 raft_consensus.cc:2781] T f611075fd74643dc84bf59471b18a55c P f8c6771f4d2747a3830a0b5681974591 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:31:10.867389 26008 leader_election.cc:302] T f611075fd74643dc84bf59471b18a55c P b067709aaa7647cb8b9efd1b4d800649 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b067709aaa7647cb8b9efd1b4d800649; no voters: 249100e6cc06451080ba73097f4b75c4, f8c6771f4d2747a3830a0b5681974591
I20260810 03:31:10.867508 26432 raft_consensus.cc:686] T f611075fd74643dc84bf59471b18a55c P f8c6771f4d2747a3830a0b5681974591 [term 1 LEADER]: Becoming Leader. State: Replica: f8c6771f4d2747a3830a0b5681974591, State: Running, Role: LEADER
I20260810 03:31:10.867812 26431 raft_consensus.cc:2726] T f611075fd74643dc84bf59471b18a55c P b067709aaa7647cb8b9efd1b4d800649 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260810 03:31:10.867995 26432 consensus_queue.cc:237] T f611075fd74643dc84bf59471b18a55c P f8c6771f4d2747a3830a0b5681974591 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b067709aaa7647cb8b9efd1b4d800649" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 41465 } } peers { permanent_uuid: "f8c6771f4d2747a3830a0b5681974591" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 33741 } } peers { permanent_uuid: "249100e6cc06451080ba73097f4b75c4" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 35683 } }
I20260810 03:31:10.872246 25663 catalog_manager.cc:5495] T f611075fd74643dc84bf59471b18a55c P f8c6771f4d2747a3830a0b5681974591 reported cstate change: term changed from 0 to 1, leader changed from <none> to f8c6771f4d2747a3830a0b5681974591 (127.27.161.3). New cstate: current_term: 1 leader_uuid: "f8c6771f4d2747a3830a0b5681974591" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b067709aaa7647cb8b9efd1b4d800649" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 41465 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f8c6771f4d2747a3830a0b5681974591" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 33741 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "249100e6cc06451080ba73097f4b75c4" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 35683 } health_report { overall_health: UNKNOWN } } }
I20260810 03:31:10.932724 26418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:31:10.927360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41034 (local address 127.27.161.3:33741)
0810 03:31:10.927571 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0810 03:31:10.927577 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:31:10.927596 (+ 19us) server_negotiation.cc:407] Connection header received
0810 03:31:10.927695 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:31:10.927699 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:31:10.927792 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:31:10.927924 (+ 132us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 03:31:10.928232 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:31:10.928986 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:31:10.930476 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:31:10.930930 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:31:10.931028 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:31:10.931506 (+ 478us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 03:31:10.932068 (+ 562us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 03:31:10.932183 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0810 03:31:10.932326 (+ 143us) server_negotiation.cc:299] Negotiation successful
0810 03:31:10.932403 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260810 03:31:10.941555 26105 raft_consensus.cc:1261] T f611075fd74643dc84bf59471b18a55c P b067709aaa7647cb8b9efd1b4d800649 [term 1 FOLLOWER]: Refusing update from remote peer f8c6771f4d2747a3830a0b5681974591: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:31:10.941574 25905 raft_consensus.cc:1261] T f611075fd74643dc84bf59471b18a55c P 249100e6cc06451080ba73097f4b75c4 [term 1 FOLLOWER]: Refusing update from remote peer f8c6771f4d2747a3830a0b5681974591: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:31:10.942525 26438 consensus_queue.cc:1035] T f611075fd74643dc84bf59471b18a55c P f8c6771f4d2747a3830a0b5681974591 [LEADER]: Connected to new peer: Peer: permanent_uuid: "249100e6cc06451080ba73097f4b75c4" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 35683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:31:10.942898 26432 consensus_queue.cc:1035] T f611075fd74643dc84bf59471b18a55c P f8c6771f4d2747a3830a0b5681974591 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b067709aaa7647cb8b9efd1b4d800649" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 41465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:31:10.953794 26441 mvcc.cc:205] Tried to move back new op lower bound from 7316818620165197824 to 7316818619878928384. Current Snapshot: MvccSnapshot[applied={T|T < 7316818620165197824}]
I20260810 03:31:10.958052 26442 mvcc.cc:205] Tried to move back new op lower bound from 7316818620165197824 to 7316818619878928384. Current Snapshot: MvccSnapshot[applied={T|T < 7316818620165197824}]
I20260810 03:31:10.958454 26443 mvcc.cc:205] Tried to move back new op lower bound from 7316818620165197824 to 7316818619878928384. Current Snapshot: MvccSnapshot[applied={T|T < 7316818620165197824 or (T in {7316818620165197824})}]
I20260810 03:31:10.967411 25663 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47198:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 03:31:10.967738 25663 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47198:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 03:31:10.971925 25663 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d4e6f3c481b6439aa7b096a70a12964a: Sending DeleteTablet for 3 replicas of tablet f611075fd74643dc84bf59471b18a55c
I20260810 03:31:10.973290 26289 tablet_service.cc:1514] Processing DeleteTablet for tablet f611075fd74643dc84bf59471b18a55c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:31:10 UTC) from {username='kudu', principal='kudu/127.27.161.62@KRBTEST.COM'} at 127.0.0.1:41032
I20260810 03:31:10.973290 26077 tablet_service.cc:1514] Processing DeleteTablet for tablet f611075fd74643dc84bf59471b18a55c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:31:10 UTC) from {username='kudu', principal='kudu/127.27.161.62@KRBTEST.COM'} at 127.0.0.1:48334
I20260810 03:31:10.973565 25876 tablet_service.cc:1514] Processing DeleteTablet for tablet f611075fd74643dc84bf59471b18a55c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:31:10 UTC) from {username='kudu', principal='kudu/127.27.161.62@KRBTEST.COM'} at 127.0.0.1:58088
I20260810 03:31:10.974972 28292 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iBH5/build/asan/bin/kudu pid 25768
I20260810 03:31:10.975013 26458 tablet_replica.cc:330] T f611075fd74643dc84bf59471b18a55c P 249100e6cc06451080ba73097f4b75c4: stopping tablet replica
I20260810 03:31:10.975473 26458 raft_consensus.cc:2229] T f611075fd74643dc84bf59471b18a55c P 249100e6cc06451080ba73097f4b75c4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 03:31:10.976025 26458 raft_consensus.cc:2258] T f611075fd74643dc84bf59471b18a55c P 249100e6cc06451080ba73097f4b75c4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:31:10.983175 26458 ts_tablet_manager.cc:1905] T f611075fd74643dc84bf59471b18a55c P 249100e6cc06451080ba73097f4b75c4: Deleting tablet data with delete state TABLET_DATA_DELETED
Aug 10 03:31:10 dist-test-slave-vpvm krb5kdc[25604](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332668, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.161.1@KRBTEST.COM
I20260810 03:31:10.991667 26456 tablet_replica.cc:330] T f611075fd74643dc84bf59471b18a55c P f8c6771f4d2747a3830a0b5681974591: stopping tablet replica
I20260810 03:31:10.996498 26458 ts_tablet_manager.cc:1918] T f611075fd74643dc84bf59471b18a55c P 249100e6cc06451080ba73097f4b75c4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 03:31:10.996687 26458 log.cc:1196] T f611075fd74643dc84bf59471b18a55c P 249100e6cc06451080ba73097f4b75c4: Deleting WAL directory at /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786332336319544-28292-0/minicluster-data/ts-0/wal/wals/f611075fd74643dc84bf59471b18a55c
I20260810 03:31:10.997263 26458 ts_tablet_manager.cc:1939] T f611075fd74643dc84bf59471b18a55c P 249100e6cc06451080ba73097f4b75c4: Deleting consensus metadata
I20260810 03:31:10.999310 26437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:31:10.977348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58102 (local address 127.27.161.1:35683)
0810 03:31:10.977569 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0810 03:31:10.977576 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:31:10.977766 (+ 190us) server_negotiation.cc:407] Connection header received
0810 03:31:10.978127 (+ 361us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:31:10.978132 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:31:10.978230 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:31:10.978312 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:31:10.980625 (+ 2313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:31:10.981834 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:31:10.983849 (+ 2015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:31:10.984462 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:31:10.984570 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:31:10.991128 (+ 6558us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:31:10.991151 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:31:10.991172 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:31:10.991234 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:31:10.995311 (+ 4077us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:31:10.996250 (+ 939us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:31:10.996261 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:31:10.996264 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:31:10.996453 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:31:10.997182 (+ 729us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:31:10.997187 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:31:10.997190 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:31:10.997782 (+ 592us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:31:10.998102 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0810 03:31:10.998695 (+ 593us) server_negotiation.cc:299] Negotiation successful
0810 03:31:10.998965 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260810 03:31:10.999420 25647 catalog_manager.cc:4841] TS 249100e6cc06451080ba73097f4b75c4 (127.27.161.1:35683): tablet f611075fd74643dc84bf59471b18a55c (table test-table [id=758d9b69e34040d5a4d32be5ca3ab8ce]) successfully deleted
I20260810 03:31:10.999889 26456 raft_consensus.cc:2229] T f611075fd74643dc84bf59471b18a55c P f8c6771f4d2747a3830a0b5681974591 [term 1 LEADER]: Raft consensus shutting down.
I20260810 03:31:11.000741 26456 raft_consensus.cc:2258] T f611075fd74643dc84bf59471b18a55c P f8c6771f4d2747a3830a0b5681974591 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:31:11.001132 25966 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:31:11.001791 26453 tablet_replica.cc:330] T f611075fd74643dc84bf59471b18a55c P b067709aaa7647cb8b9efd1b4d800649: stopping tablet replica
I20260810 03:31:11.002311 26453 raft_consensus.cc:2229] T f611075fd74643dc84bf59471b18a55c P b067709aaa7647cb8b9efd1b4d800649 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 03:31:11.002925 26453 raft_consensus.cc:2258] T f611075fd74643dc84bf59471b18a55c P b067709aaa7647cb8b9efd1b4d800649 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:31:11.003276 26456 ts_tablet_manager.cc:1905] T f611075fd74643dc84bf59471b18a55c P f8c6771f4d2747a3830a0b5681974591: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:31:11.008769 26453 ts_tablet_manager.cc:1905] T f611075fd74643dc84bf59471b18a55c P b067709aaa7647cb8b9efd1b4d800649: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:31:11.011552 26456 ts_tablet_manager.cc:1918] T f611075fd74643dc84bf59471b18a55c P f8c6771f4d2747a3830a0b5681974591: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 03:31:11.011816 26456 log.cc:1196] T f611075fd74643dc84bf59471b18a55c P f8c6771f4d2747a3830a0b5681974591: Deleting WAL directory at /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786332336319544-28292-0/minicluster-data/ts-2/wal/wals/f611075fd74643dc84bf59471b18a55c
I20260810 03:31:11.012472 26456 ts_tablet_manager.cc:1939] T f611075fd74643dc84bf59471b18a55c P f8c6771f4d2747a3830a0b5681974591: Deleting consensus metadata
I20260810 03:31:11.014554 25647 catalog_manager.cc:4841] TS f8c6771f4d2747a3830a0b5681974591 (127.27.161.3:33741): tablet f611075fd74643dc84bf59471b18a55c (table test-table [id=758d9b69e34040d5a4d32be5ca3ab8ce]) successfully deleted
I20260810 03:31:11.016253 26453 ts_tablet_manager.cc:1918] T f611075fd74643dc84bf59471b18a55c P b067709aaa7647cb8b9efd1b4d800649: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 03:31:11.016448 26453 log.cc:1196] T f611075fd74643dc84bf59471b18a55c P b067709aaa7647cb8b9efd1b4d800649: Deleting WAL directory at /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786332336319544-28292-0/minicluster-data/ts-1/wal/wals/f611075fd74643dc84bf59471b18a55c
I20260810 03:31:11.017031 26453 ts_tablet_manager.cc:1939] T f611075fd74643dc84bf59471b18a55c P b067709aaa7647cb8b9efd1b4d800649: Deleting consensus metadata
I20260810 03:31:11.019116 25648 catalog_manager.cc:4841] TS b067709aaa7647cb8b9efd1b4d800649 (127.27.161.2:41465): tablet f611075fd74643dc84bf59471b18a55c (table test-table [id=758d9b69e34040d5a4d32be5ca3ab8ce]) successfully deleted
I20260810 03:31:11.098206 28292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iBH5/build/asan/bin/kudu with pid 25768
I20260810 03:31:11.131847 28292 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iBH5/build/asan/bin/kudu pid 25985
Aug 10 03:31:11 dist-test-slave-vpvm krb5kdc[25604](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332668, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.161.2@KRBTEST.COM
I20260810 03:31:11.149910 26415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:31:11.132691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48344 (local address 127.27.161.2:41465)
0810 03:31:11.132869 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0810 03:31:11.132873 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:31:11.132894 (+ 21us) server_negotiation.cc:407] Connection header received
0810 03:31:11.133057 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:31:11.133060 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:31:11.133130 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:31:11.133226 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:31:11.135097 (+ 1871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:31:11.135847 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:31:11.137775 (+ 1928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:31:11.138239 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:31:11.138340 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:31:11.143960 (+ 5620us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:31:11.143986 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:31:11.144007 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:31:11.144055 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:31:11.147031 (+ 2976us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:31:11.147905 (+ 874us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:31:11.147912 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:31:11.147915 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:31:11.148059 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:31:11.148438 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:31:11.148441 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:31:11.148443 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:31:11.148851 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:31:11.149076 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0810 03:31:11.149428 (+ 352us) server_negotiation.cc:299] Negotiation successful
0810 03:31:11.149618 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260810 03:31:11.150712 26167 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:31:11.245767 28292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iBH5/build/asan/bin/kudu with pid 25985
I20260810 03:31:11.272282 28292 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iBH5/build/asan/bin/kudu pid 26180
Aug 10 03:31:11 dist-test-slave-vpvm krb5kdc[25604](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332668, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.161.3@KRBTEST.COM
I20260810 03:31:11.296566 26418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:31:11.273125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41036 (local address 127.27.161.3:33741)
0810 03:31:11.273306 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0810 03:31:11.273311 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:31:11.273330 (+ 19us) server_negotiation.cc:407] Connection header received
0810 03:31:11.273608 (+ 278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:31:11.273613 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:31:11.273688 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:31:11.273781 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:31:11.275668 (+ 1887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:31:11.276492 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:31:11.278423 (+ 1931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:31:11.278986 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:31:11.279126 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:31:11.288637 (+ 9511us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:31:11.288663 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:31:11.288687 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:31:11.288750 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:31:11.293043 (+ 4293us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:31:11.293929 (+ 886us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:31:11.293941 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:31:11.293945 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:31:11.294185 (+ 240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:31:11.294678 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:31:11.294683 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:31:11.294687 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:31:11.295248 (+ 561us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:31:11.295646 (+ 398us) server_negotiation.cc:1035] Waiting for connection context
0810 03:31:11.295908 (+ 262us) server_negotiation.cc:299] Negotiation successful
0810 03:31:11.296177 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260810 03:31:11.297431 26379 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:31:11.422031 28292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iBH5/build/asan/bin/kudu with pid 26180
I20260810 03:31:11.449906 28292 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iBH5/build/asan/bin/kudu pid 25624
I20260810 03:31:11.451081 25717 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:31:11.567551 28292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iBH5/build/asan/bin/kudu with pid 25624
2026-08-10T03:31:11Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (5027 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-08-10T03:31:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T03:31:11Z Disabled control of system clock
I20260810 03:31:11.668887 28292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iBH5/build/asan/bin/kudu
/tmp/dist-test-task24iBH5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786332336319544-28292-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786332336319544-28292-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786332336319544-28292-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786332336319544-28292-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.161.62:39261
--webserver_interface=127.27.161.62
--webserver_port=0
--builtin_ntp_servers=127.27.161.20:38079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.161.62:39261
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260810 03:31:12.030026 26472 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:31:12.030375 26472 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:31:12.030462 26472 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:31:12.038424 26472 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:31:12.038561 26472 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:31:12.038599 26472 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:31:12.038674 26472 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:31:12.038719 26472 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:31:12.048442 26472 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.161.20:38079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786332336319544-28292-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786332336319544-28292-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.161.62:39261
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.161.62:39261
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786332336319544-28292-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.161.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786332336319544-28292-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:31:12.049858 26472 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:31:12.051857 26472 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:31:12.064059 26479 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:31:12.064217 26478 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:31:12.064194 26482 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:31:12.064998 26472 server_base.cc:1034] running on GCE node
I20260810 03:31:12.065783 26472 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:31:12.067325 26472 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:31:12.068528 26472 hybrid_clock.cc:648] HybridClock initialized: now 1786332672068475 us; error 43 us; skew 500 ppm
I20260810 03:31:12.068902 26472 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 03:31:12.071280 26472 webserver.cc:460] Webserver started at http://127.27.161.62:42649/ using document root <none> and password file <none>
I20260810 03:31:12.071998 26472 fs_manager.cc:359] Metadata directory not provided
I20260810 03:31:12.072098 26472 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:31:12.072438 26472 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:31:12.075585 26472 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786332336319544-28292-0/minicluster-data/master-0/data/instance:
uuid: "1efa32a650084dc98a1526c1eabf83ed"
format_stamp: "Formatted at 2026-08-10 03:31:12 on dist-test-slave-vpvm"
server_key: "6ff460b915573e318c2e3d8d47c73c6c"
server_key_iv: "4f8d3b0495382626b99a332af2605cbc"
server_key_version: "encryptionkey@0"
I20260810 03:31:12.076468 26472 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786332336319544-28292-0/minicluster-data/master-0/wal/instance:
uuid: "1efa32a650084dc98a1526c1eabf83ed"
format_stamp: "Formatted at 2026-08-10 03:31:12 on dist-test-slave-vpvm"
server_key: "6ff460b915573e318c2e3d8d47c73c6c"
server_key_iv: "4f8d3b0495382626b99a332af2605cbc"
server_key_version: "encryptionkey@0"
I20260810 03:31:12.082262 26472 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260810 03:31:12.086266 26490 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:31:12.087708 26472 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 03:31:12.087888 26472 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786332336319544-28292-0/minicluster-data/master-0/data,/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786332336319544-28292-0/minicluster-data/master-0/wal
uuid: "1efa32a650084dc98a1526c1eabf83ed"
format_stamp: "Formatted at 2026-08-10 03:31:12 on dist-test-slave-vpvm"
server_key: "6ff460b915573e318c2e3d8d47c73c6c"
server_key_iv: "4f8d3b0495382626b99a332af2605cbc"
server_key_version: "encryptionkey@0"
I20260810 03:31:12.088126 26472 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786332336319544-28292-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786332336319544-28292-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786332336319544-28292-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:31:12.112142 26472 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:31:12.113116 26472 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:31:12.113428 26472 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:31:12.131925 26472 rpc_server.cc:307] RPC server started. Bound to: 127.27.161.62:39261
I20260810 03:31:12.131933 26576 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.161.62:39261 every 8 connection(s)
I20260810 03:31:12.133397 26472 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786332336319544-28292-0/minicluster-data/master-0/data/info.pb
I20260810 03:31:12.137423 26577 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:31:12.143210 28292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iBH5/build/asan/bin/kudu as pid 26472
I20260810 03:31:12.143399 28292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786332336319544-28292-0/minicluster-data/master-0/wal/instance
I20260810 03:31:12.143887 28292 external_mini_cluster.cc:1428] Setting key 45de4a933f7d141ba60417a76ded1646
I20260810 03:31:12.150254 26577 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1efa32a650084dc98a1526c1eabf83ed: Bootstrap starting.
I20260810 03:31:12.154290 26577 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1efa32a650084dc98a1526c1eabf83ed: Neither blocks nor log segments found. Creating new log.
I20260810 03:31:12.155424 26577 log.cc:824] T 00000000000000000000000000000000 P 1efa32a650084dc98a1526c1eabf83ed: Log is configured to *not* fsync() on all Append() calls
I20260810 03:31:12.159158 26577 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1efa32a650084dc98a1526c1eabf83ed: No bootstrap required, opened a new log
I20260810 03:31:12.160665 26583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:31:12.146095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49414 (local address 127.27.161.62:39261)
0810 03:31:12.147375 (+ 1280us) server_negotiation.cc:206] Beginning negotiation
0810 03:31:12.147392 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0810 03:31:12.147452 (+ 60us) server_negotiation.cc:407] Connection header received
0810 03:31:12.149913 (+ 2461us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:31:12.149945 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:31:12.150614 (+ 669us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:31:12.151061 (+ 447us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:31:12.151617 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:31:12.153607 (+ 1990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:31:12.155142 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:31:12.156227 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:31:12.156373 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:31:12.157712 (+ 1339us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:31:12.157724 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:31:12.157743 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:31:12.158522 (+ 779us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:31:12.158634 (+ 112us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:31:12.158662 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:31:12.158788 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0810 03:31:12.158989 (+ 201us) server_negotiation.cc:299] Negotiation successful
0810 03:31:12.159353 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":653,"thread_start_us":394,"threads_started":1}
I20260810 03:31:12.167219 26577 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1efa32a650084dc98a1526c1eabf83ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1efa32a650084dc98a1526c1eabf83ed" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 39261 } }
I20260810 03:31:12.167614 26577 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1efa32a650084dc98a1526c1eabf83ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:31:12.167699 26577 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1efa32a650084dc98a1526c1eabf83ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1efa32a650084dc98a1526c1eabf83ed, State: Initialized, Role: FOLLOWER
I20260810 03:31:12.168238 26577 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1efa32a650084dc98a1526c1eabf83ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1efa32a650084dc98a1526c1eabf83ed" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 39261 } }
I20260810 03:31:12.168470 26577 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1efa32a650084dc98a1526c1eabf83ed [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 03:31:12.168617 26577 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1efa32a650084dc98a1526c1eabf83ed [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 03:31:12.168788 26577 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1efa32a650084dc98a1526c1eabf83ed [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:31:12.171120 26577 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1efa32a650084dc98a1526c1eabf83ed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1efa32a650084dc98a1526c1eabf83ed" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 39261 } }
I20260810 03:31:12.171662 26577 leader_election.cc:302] T 00000000000000000000000000000000 P 1efa32a650084dc98a1526c1eabf83ed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1efa32a650084dc98a1526c1eabf83ed; no voters:
I20260810 03:31:12.172255 26577 leader_election.cc:288] T 00000000000000000000000000000000 P 1efa32a650084dc98a1526c1eabf83ed [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 03:31:12.172446 26586 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1efa32a650084dc98a1526c1eabf83ed [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:31:12.173650 26586 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1efa32a650084dc98a1526c1eabf83ed [term 1 LEADER]: Becoming Leader. State: Replica: 1efa32a650084dc98a1526c1eabf83ed, State: Running, Role: LEADER
I20260810 03:31:12.174243 26586 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1efa32a650084dc98a1526c1eabf83ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1efa32a650084dc98a1526c1eabf83ed" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 39261 } }
I20260810 03:31:12.174587 26577 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1efa32a650084dc98a1526c1eabf83ed [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:31:12.177170 26589 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1efa32a650084dc98a1526c1eabf83ed [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1efa32a650084dc98a1526c1eabf83ed. Latest consensus state: current_term: 1 leader_uuid: "1efa32a650084dc98a1526c1eabf83ed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1efa32a650084dc98a1526c1eabf83ed" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 39261 } } }
I20260810 03:31:12.177627 26589 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1efa32a650084dc98a1526c1eabf83ed [sys.catalog]: This master's current role is: LEADER
I20260810 03:31:12.177148 26588 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1efa32a650084dc98a1526c1eabf83ed [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1efa32a650084dc98a1526c1eabf83ed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1efa32a650084dc98a1526c1eabf83ed" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 39261 } } }
I20260810 03:31:12.177744 26588 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1efa32a650084dc98a1526c1eabf83ed [sys.catalog]: This master's current role is: LEADER
I20260810 03:31:12.178803 26598 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 03:31:12.184777 26598 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 03:31:12.193872 26598 catalog_manager.cc:1347] Generated new cluster ID: 2c56876945a045878f06999c3b174987
I20260810 03:31:12.194010 26598 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 03:31:12.216692 26598 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 03:31:12.218128 26598 catalog_manager.cc:1504] Loading token signing keys...
I20260810 03:31:12.227042 26598 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1efa32a650084dc98a1526c1eabf83ed: Generated new TSK 0
I20260810 03:31:12.228291 26598 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 03:31:12.244863 28292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iBH5/build/asan/bin/kudu
/tmp/dist-test-task24iBH5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786332336319544-28292-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786332336319544-28292-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786332336319544-28292-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786332336319544-28292-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.161.1:0
--local_ip_for_outbound_sockets=127.27.161.1
--webserver_interface=127.27.161.1
--webserver_port=0
--tserver_master_addrs=127.27.161.62:39261
--builtin_ntp_servers=127.27.161.20:38079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260810 03:31:12.599563 26616 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:31:12.599874 26616 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:31:12.600004 26616 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:31:12.607381 26616 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:31:12.607497 26616 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:31:12.607643 26616 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.161.1
I20260810 03:31:12.616988 26616 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.161.20:38079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786332336319544-28292-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786332336319544-28292-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.161.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786332336319544-28292-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.161.1
--webserver_port=0
--tserver_master_addrs=127.27.161.62:39261
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.161.1
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786332336319544-28292-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:31:12.618360 26616 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:31:12.620186 26616 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:31:12.633064 26627 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:31:12.633173 26624 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:31:12.633212 26623 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:31:12.634382 26616 server_base.cc:1034] running on GCE node
I20260810 03:31:12.634992 26616 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:31:12.635910 26616 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:31:12.637115 26616 hybrid_clock.cc:648] HybridClock initialized: now 1786332672637071 us; error 33 us; skew 500 ppm
I20260810 03:31:12.637416 26616 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 03:31:12.639573 26616 webserver.cc:460] Webserver started at http://127.27.161.1:38219/ using document root <none> and password file <none>
I20260810 03:31:12.640251 26616 fs_manager.cc:359] Metadata directory not provided
I20260810 03:31:12.640336 26616 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:31:12.640595 26616 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:31:12.643467 26616 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786332336319544-28292-0/minicluster-data/ts-0/data/instance:
uuid: "cd31b5efc51f40a4bea1e4497d2d2b4a"
format_stamp: "Formatted at 2026-08-10 03:31:12 on dist-test-slave-vpvm"
server_key: "fc58b4e199960331e60c07c172c8fd76"
server_key_iv: "91ea85def7926da798471575cfd5161f"
server_key_version: "encryptionkey@0"
I20260810 03:31:12.644223 26616 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786332336319544-28292-0/minicluster-data/ts-0/wal/instance:
uuid: "cd31b5efc51f40a4bea1e4497d2d2b4a"
format_stamp: "Formatted at 2026-08-10 03:31:12 on dist-test-slave-vpvm"
server_key: "fc58b4e199960331e60c07c172c8fd76"
server_key_iv: "91ea85def7926da798471575cfd5161f"
server_key_version: "encryptionkey@0"
I20260810 03:31:12.649327 26616 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260810 03:31:12.652822 26637 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:31:12.654172 26616 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 03:31:12.654323 26616 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786332336319544-28292-0/minicluster-data/ts-0/data,/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786332336319544-28292-0/minicluster-data/ts-0/wal
uuid: "cd31b5efc51f40a4bea1e4497d2d2b4a"
format_stamp: "Formatted at 2026-08-10 03:31:12 on dist-test-slave-vpvm"
server_key: "fc58b4e199960331e60c07c172c8fd76"
server_key_iv: "91ea85def7926da798471575cfd5161f"
server_key_version: "encryptionkey@0"
I20260810 03:31:12.654528 26616 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786332336319544-28292-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786332336319544-28292-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786332336319544-28292-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:31:12.684646 26616 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:31:12.685516 26616 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:31:12.685799 26616 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:31:12.686866 26616 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:31:12.688367 26616 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:31:12.688458 26616 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:31:12.688567 26616 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:31:12.688643 26616 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:31:12.720260 26616 rpc_server.cc:307] RPC server started. Bound to: 127.27.161.1:38397
I20260810 03:31:12.720338 26801 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.161.1:38397 every 8 connection(s)
I20260810 03:31:12.721614 26616 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786332336319544-28292-0/minicluster-data/ts-0/data/info.pb
I20260810 03:31:12.731076 28292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iBH5/build/asan/bin/kudu as pid 26616
I20260810 03:31:12.731246 28292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786332336319544-28292-0/minicluster-data/ts-0/wal/instance
I20260810 03:31:12.731720 28292 external_mini_cluster.cc:1428] Setting key d6729ecbb3bc291bcc262deb58e2d75c
I20260810 03:31:12.734550 26804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:31:12.724235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.161.1:51641 (local address 127.27.161.62:39261)
0810 03:31:12.724673 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0810 03:31:12.724678 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:31:12.725902 (+ 1224us) server_negotiation.cc:407] Connection header received
0810 03:31:12.727837 (+ 1935us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:31:12.727844 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:31:12.727929 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:31:12.728030 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:31:12.729196 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:31:12.729932 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:31:12.731661 (+ 1729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:31:12.732104 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:31:12.732239 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:31:12.733665 (+ 1426us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:31:12.733670 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:31:12.733673 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:31:12.733692 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:31:12.733734 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:31:12.733745 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:31:12.733823 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0810 03:31:12.734076 (+ 253us) server_negotiation.cc:299] Negotiation successful
0810 03:31:12.734194 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":214,"threads_started":1}
I20260810 03:31:12.734542 28292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iBH5/build/asan/bin/kudu
/tmp/dist-test-task24iBH5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786332336319544-28292-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786332336319544-28292-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786332336319544-28292-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786332336319544-28292-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.161.2:0
--local_ip_for_outbound_sockets=127.27.161.2
--webserver_interface=127.27.161.2
--webserver_port=0
--tserver_master_addrs=127.27.161.62:39261
--builtin_ntp_servers=127.27.161.20:38079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260810 03:31:12.735670 26806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:31:12.724764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:39261 (local address 127.27.161.1:51641)
0810 03:31:12.725728 (+ 964us) negotiation.cc:107] Waiting for socket to connect
0810 03:31:12.725775 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0810 03:31:12.727541 (+ 1766us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:31:12.728293 (+ 752us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:31:12.728302 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:31:12.728396 (+ 94us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:31:12.729005 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:31:12.729024 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:31:12.730116 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:31:12.730121 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:31:12.731507 (+ 1386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:31:12.731521 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:31:12.732355 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:31:12.732363 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:31:12.732605 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:31:12.733326 (+ 721us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 03:31:12.733329 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:31:12.733373 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 03:31:12.733462 (+ 89us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:31:12.733838 (+ 376us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:31:12.733852 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:31:12.733860 (+ 8us) client_negotiation.cc:769] Sending connection context
0810 03:31:12.734060 (+ 200us) client_negotiation.cc:240] Negotiation successful
0810 03:31:12.734498 (+ 438us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":502,"thread_start_us":310,"threads_started":1}
I20260810 03:31:12.737205 26802 heartbeater.cc:347] Connected to a master server at 127.27.161.62:39261
I20260810 03:31:12.737519 26802 heartbeater.cc:464] Registering TS with master...
I20260810 03:31:12.738365 26802 heartbeater.cc:511] Master 127.27.161.62:39261 requested a full tablet report, sending...
I20260810 03:31:12.740486 26517 ts_manager.cc:194] Registered new tserver with Master: cd31b5efc51f40a4bea1e4497d2d2b4a (127.27.161.1:38397)
I20260810 03:31:12.741990 26517 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.161.1:51641
W20260810 03:31:13.082101 26807 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:31:13.082440 26807 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:31:13.082581 26807 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:31:13.089977 26807 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:31:13.090092 26807 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:31:13.090240 26807 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.161.2
I20260810 03:31:13.099663 26807 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.161.20:38079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786332336319544-28292-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786332336319544-28292-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.161.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786332336319544-28292-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.161.2
--webserver_port=0
--tserver_master_addrs=127.27.161.62:39261
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.161.2
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786332336319544-28292-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:31:13.101033 26807 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:31:13.102934 26807 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:31:13.116557 26813 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:31:13.116598 26817 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:31:13.116521 26815 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:31:13.117698 26807 server_base.cc:1034] running on GCE node
I20260810 03:31:13.118422 26807 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:31:13.119438 26807 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:31:13.120677 26807 hybrid_clock.cc:648] HybridClock initialized: now 1786332673120612 us; error 48 us; skew 500 ppm
I20260810 03:31:13.121021 26807 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 03:31:13.123443 26807 webserver.cc:460] Webserver started at http://127.27.161.2:32841/ using document root <none> and password file <none>
I20260810 03:31:13.124168 26807 fs_manager.cc:359] Metadata directory not provided
I20260810 03:31:13.124269 26807 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:31:13.124588 26807 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:31:13.127591 26807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786332336319544-28292-0/minicluster-data/ts-1/data/instance:
uuid: "344433e16c7043da9ea03360fa7ff176"
format_stamp: "Formatted at 2026-08-10 03:31:13 on dist-test-slave-vpvm"
server_key: "0b501640c3d24b1ea0a56b6cd554ef01"
server_key_iv: "ee097c9124ca7604a2f8f20ca6d6b364"
server_key_version: "encryptionkey@0"
I20260810 03:31:13.128404 26807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786332336319544-28292-0/minicluster-data/ts-1/wal/instance:
uuid: "344433e16c7043da9ea03360fa7ff176"
format_stamp: "Formatted at 2026-08-10 03:31:13 on dist-test-slave-vpvm"
server_key: "0b501640c3d24b1ea0a56b6cd554ef01"
server_key_iv: "ee097c9124ca7604a2f8f20ca6d6b364"
server_key_version: "encryptionkey@0"
I20260810 03:31:13.133924 26807 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260810 03:31:13.137753 26830 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:31:13.139397 26807 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 03:31:13.139657 26807 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786332336319544-28292-0/minicluster-data/ts-1/data,/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786332336319544-28292-0/minicluster-data/ts-1/wal
uuid: "344433e16c7043da9ea03360fa7ff176"
format_stamp: "Formatted at 2026-08-10 03:31:13 on dist-test-slave-vpvm"
server_key: "0b501640c3d24b1ea0a56b6cd554ef01"
server_key_iv: "ee097c9124ca7604a2f8f20ca6d6b364"
server_key_version: "encryptionkey@0"
I20260810 03:31:13.139928 26807 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786332336319544-28292-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786332336319544-28292-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786332336319544-28292-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:31:13.188338 26807 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:31:13.189283 26807 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:31:13.189585 26807 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:31:13.190676 26807 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:31:13.192293 26807 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:31:13.192399 26807 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:31:13.192505 26807 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:31:13.192574 26807 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:31:13.225301 26807 rpc_server.cc:307] RPC server started. Bound to: 127.27.161.2:41791
I20260810 03:31:13.225351 27004 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.161.2:41791 every 8 connection(s)
I20260810 03:31:13.226696 26807 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786332336319544-28292-0/minicluster-data/ts-1/data/info.pb
I20260810 03:31:13.230099 28292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iBH5/build/asan/bin/kudu as pid 26807
I20260810 03:31:13.230289 28292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786332336319544-28292-0/minicluster-data/ts-1/wal/instance
I20260810 03:31:13.230911 28292 external_mini_cluster.cc:1428] Setting key 217a3c6ae9f861348a8f4146ff7ec52b
I20260810 03:31:13.233980 28292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iBH5/build/asan/bin/kudu
/tmp/dist-test-task24iBH5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786332336319544-28292-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786332336319544-28292-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786332336319544-28292-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786332336319544-28292-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.161.3:0
--local_ip_for_outbound_sockets=127.27.161.3
--webserver_interface=127.27.161.3
--webserver_port=0
--tserver_master_addrs=127.27.161.62:39261
--builtin_ntp_servers=127.27.161.20:38079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260810 03:31:13.240470 26804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:31:13.229725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.161.2:55329 (local address 127.27.161.62:39261)
0810 03:31:13.229897 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0810 03:31:13.229902 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:31:13.231693 (+ 1791us) server_negotiation.cc:407] Connection header received
0810 03:31:13.233694 (+ 2001us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:31:13.233699 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:31:13.233785 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:31:13.233907 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:31:13.235061 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:31:13.236050 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:31:13.237790 (+ 1740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:31:13.238344 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:31:13.238469 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:31:13.239741 (+ 1272us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:31:13.239747 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:31:13.239751 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:31:13.239774 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:31:13.239814 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:31:13.239828 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:31:13.239916 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0810 03:31:13.240085 (+ 169us) server_negotiation.cc:299] Negotiation successful
0810 03:31:13.240183 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260810 03:31:13.241465 27009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:31:13.230461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:39261 (local address 127.27.161.2:55329)
0810 03:31:13.231482 (+ 1021us) negotiation.cc:107] Waiting for socket to connect
0810 03:31:13.231535 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0810 03:31:13.233418 (+ 1883us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:31:13.234171 (+ 753us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:31:13.234184 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:31:13.234286 (+ 102us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:31:13.234878 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:31:13.234896 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:31:13.236226 (+ 1330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:31:13.236231 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:31:13.237639 (+ 1408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:31:13.237654 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:31:13.238538 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:31:13.238544 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:31:13.238752 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:31:13.239416 (+ 664us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 03:31:13.239419 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:31:13.239460 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 03:31:13.239543 (+ 83us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:31:13.239910 (+ 367us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:31:13.239916 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:31:13.239922 (+ 6us) client_negotiation.cc:769] Sending connection context
0810 03:31:13.240064 (+ 142us) client_negotiation.cc:240] Negotiation successful
0810 03:31:13.240407 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":561,"thread_start_us":321,"threads_started":1}
I20260810 03:31:13.242892 27005 heartbeater.cc:347] Connected to a master server at 127.27.161.62:39261
I20260810 03:31:13.243212 27005 heartbeater.cc:464] Registering TS with master...
I20260810 03:31:13.244074 27005 heartbeater.cc:511] Master 127.27.161.62:39261 requested a full tablet report, sending...
I20260810 03:31:13.245543 26517 ts_manager.cc:194] Registered new tserver with Master: 344433e16c7043da9ea03360fa7ff176 (127.27.161.2:41791)
I20260810 03:31:13.246400 26517 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.161.2:55329
W20260810 03:31:13.592676 27011 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:31:13.592981 27011 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:31:13.593101 27011 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:31:13.600533 27011 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:31:13.600653 27011 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:31:13.600760 27011 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.161.3
I20260810 03:31:13.610457 27011 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.161.20:38079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786332336319544-28292-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786332336319544-28292-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.161.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786332336319544-28292-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.161.3
--webserver_port=0
--tserver_master_addrs=127.27.161.62:39261
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.161.3
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786332336319544-28292-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:31:13.611747 27011 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:31:13.613503 27011 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:31:13.627115 27019 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:31:13.627038 27023 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:31:13.627094 27020 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:31:13.628051 27011 server_base.cc:1034] running on GCE node
I20260810 03:31:13.628700 27011 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:31:13.629834 27011 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:31:13.631063 27011 hybrid_clock.cc:648] HybridClock initialized: now 1786332673631014 us; error 50 us; skew 500 ppm
I20260810 03:31:13.631402 27011 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 03:31:13.633672 27011 webserver.cc:460] Webserver started at http://127.27.161.3:45353/ using document root <none> and password file <none>
I20260810 03:31:13.634438 27011 fs_manager.cc:359] Metadata directory not provided
I20260810 03:31:13.634543 27011 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:31:13.634869 27011 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:31:13.637861 27011 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786332336319544-28292-0/minicluster-data/ts-2/data/instance:
uuid: "9b3d9d39f2b746e4830ff9d0f3ef6457"
format_stamp: "Formatted at 2026-08-10 03:31:13 on dist-test-slave-vpvm"
server_key: "5960c407f912e0930bde8e9b469940cb"
server_key_iv: "7f4acea683323539f6025d2a5dc2b867"
server_key_version: "encryptionkey@0"
I20260810 03:31:13.638753 27011 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786332336319544-28292-0/minicluster-data/ts-2/wal/instance:
uuid: "9b3d9d39f2b746e4830ff9d0f3ef6457"
format_stamp: "Formatted at 2026-08-10 03:31:13 on dist-test-slave-vpvm"
server_key: "5960c407f912e0930bde8e9b469940cb"
server_key_iv: "7f4acea683323539f6025d2a5dc2b867"
server_key_version: "encryptionkey@0"
I20260810 03:31:13.644290 27011 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260810 03:31:13.648159 27034 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:31:13.649662 27011 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 03:31:13.649842 27011 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786332336319544-28292-0/minicluster-data/ts-2/data,/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786332336319544-28292-0/minicluster-data/ts-2/wal
uuid: "9b3d9d39f2b746e4830ff9d0f3ef6457"
format_stamp: "Formatted at 2026-08-10 03:31:13 on dist-test-slave-vpvm"
server_key: "5960c407f912e0930bde8e9b469940cb"
server_key_iv: "7f4acea683323539f6025d2a5dc2b867"
server_key_version: "encryptionkey@0"
I20260810 03:31:13.650106 27011 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786332336319544-28292-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786332336319544-28292-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786332336319544-28292-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:31:13.672708 27011 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:31:13.673605 27011 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:31:13.673909 27011 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:31:13.675055 27011 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:31:13.676689 27011 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:31:13.676789 27011 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:31:13.676903 27011 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:31:13.676975 27011 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:31:13.709259 27011 rpc_server.cc:307] RPC server started. Bound to: 127.27.161.3:38417
I20260810 03:31:13.709331 27213 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.161.3:38417 every 8 connection(s)
I20260810 03:31:13.710661 27011 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786332336319544-28292-0/minicluster-data/ts-2/data/info.pb
I20260810 03:31:13.720511 28292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iBH5/build/asan/bin/kudu as pid 27011
I20260810 03:31:13.720724 28292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786332336319544-28292-0/minicluster-data/ts-2/wal/instance
I20260810 03:31:13.721299 28292 external_mini_cluster.cc:1428] Setting key 734aee2dd338cab921f4a4b16cb36ae1
I20260810 03:31:13.725265 26804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:31:13.713495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.161.3:44613 (local address 127.27.161.62:39261)
0810 03:31:13.713679 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0810 03:31:13.713685 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:31:13.715287 (+ 1602us) server_negotiation.cc:407] Connection header received
0810 03:31:13.717175 (+ 1888us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:31:13.717182 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:31:13.717275 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:31:13.717369 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:31:13.718763 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:31:13.719544 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:31:13.721926 (+ 2382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:31:13.722527 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:31:13.722671 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:31:13.724404 (+ 1733us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:31:13.724410 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:31:13.724414 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:31:13.724438 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:31:13.724480 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:31:13.724497 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:31:13.724575 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0810 03:31:13.724911 (+ 336us) server_negotiation.cc:299] Negotiation successful
0810 03:31:13.725002 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260810 03:31:13.726516 27221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:31:13.714080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:39261 (local address 127.27.161.3:44613)
0810 03:31:13.715063 (+ 983us) negotiation.cc:107] Waiting for socket to connect
0810 03:31:13.715121 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0810 03:31:13.716848 (+ 1727us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:31:13.717634 (+ 786us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:31:13.717643 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:31:13.717744 (+ 101us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:31:13.718538 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:31:13.718566 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:31:13.719737 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:31:13.719742 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:31:13.721738 (+ 1996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:31:13.721756 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:31:13.722804 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:31:13.722812 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:31:13.723137 (+ 325us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:31:13.723977 (+ 840us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 03:31:13.723981 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:31:13.724036 (+ 55us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 03:31:13.724158 (+ 122us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:31:13.724676 (+ 518us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:31:13.724687 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:31:13.724695 (+ 8us) client_negotiation.cc:769] Sending connection context
0810 03:31:13.724885 (+ 190us) client_negotiation.cc:240] Negotiation successful
0810 03:31:13.725303 (+ 418us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":526,"thread_start_us":300,"threads_started":1}
I20260810 03:31:13.728070 27218 heartbeater.cc:347] Connected to a master server at 127.27.161.62:39261
I20260810 03:31:13.728440 27218 heartbeater.cc:464] Registering TS with master...
I20260810 03:31:13.729274 27218 heartbeater.cc:511] Master 127.27.161.62:39261 requested a full tablet report, sending...
I20260810 03:31:13.730835 26517 ts_manager.cc:194] Registered new tserver with Master: 9b3d9d39f2b746e4830ff9d0f3ef6457 (127.27.161.3:38417)
I20260810 03:31:13.731648 26517 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.161.3:44613
I20260810 03:31:13.736996 28292 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 03:31:13.744946 26802 heartbeater.cc:503] Master 127.27.161.62:39261 was elected leader, sending a full tablet report...
I20260810 03:31:13.747694 26804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:31:13.741701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49426 (local address 127.27.161.62:39261)
0810 03:31:13.741980 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0810 03:31:13.741989 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 03:31:13.742329 (+ 340us) server_negotiation.cc:407] Connection header received
0810 03:31:13.742522 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:31:13.742526 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:31:13.742599 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:31:13.742686 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:31:13.743196 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:31:13.743928 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:31:13.745365 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:31:13.745820 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:31:13.745936 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:31:13.747002 (+ 1066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:31:13.747007 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:31:13.747010 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:31:13.747029 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:31:13.747071 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:31:13.747083 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:31:13.747160 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0810 03:31:13.747289 (+ 129us) server_negotiation.cc:299] Negotiation successful
0810 03:31:13.747391 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260810 03:31:13.751556 26517 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49426:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 03:31:13.758713 26517 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 03:31:13.797626 27246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:31:13.773112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45800 (local address 127.27.161.2:41791)
0810 03:31:13.785114 (+ 12002us) server_negotiation.cc:206] Beginning negotiation
0810 03:31:13.785138 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0810 03:31:13.789088 (+ 3950us) server_negotiation.cc:407] Connection header received
0810 03:31:13.789214 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:31:13.789222 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:31:13.789493 (+ 271us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:31:13.789715 (+ 222us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:31:13.790730 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:31:13.792068 (+ 1338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:31:13.795046 (+ 2978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:31:13.796362 (+ 1316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:31:13.796489 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:31:13.796757 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0810 03:31:13.797086 (+ 329us) server_negotiation.cc:299] Negotiation successful
0810 03:31:13.797268 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11776,"thread_start_us":11657,"threads_started":1}
I20260810 03:31:13.797616 27243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:31:13.772038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.2:41791 (local address 127.0.0.1:45800)
0810 03:31:13.788162 (+ 16124us) negotiation.cc:107] Waiting for socket to connect
0810 03:31:13.788185 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0810 03:31:13.788338 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:31:13.789903 (+ 1565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:31:13.789911 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:31:13.789941 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:31:13.790455 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:31:13.790470 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:31:13.792220 (+ 1750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:31:13.792226 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:31:13.794804 (+ 2578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:31:13.794817 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:31:13.796602 (+ 1785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:31:13.796609 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:31:13.796834 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:31:13.796887 (+ 53us) client_negotiation.cc:769] Sending connection context
0810 03:31:13.796993 (+ 106us) client_negotiation.cc:240] Negotiation successful
0810 03:31:13.797230 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15983,"mutex_wait_us":156,"thread_start_us":17356,"threads_started":1}
I20260810 03:31:13.797935 27245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:31:13.772730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60256 (local address 127.27.161.3:38417)
0810 03:31:13.789930 (+ 17200us) server_negotiation.cc:206] Beginning negotiation
0810 03:31:13.789939 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 03:31:13.789993 (+ 54us) server_negotiation.cc:407] Connection header received
0810 03:31:13.790069 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:31:13.790075 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:31:13.790278 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:31:13.790468 (+ 190us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:31:13.791156 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:31:13.792676 (+ 1520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:31:13.794730 (+ 2054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:31:13.795949 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:31:13.796094 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:31:13.796408 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0810 03:31:13.797356 (+ 948us) server_negotiation.cc:299] Negotiation successful
0810 03:31:13.797536 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17017,"thread_start_us":16855,"threads_started":1}
I20260810 03:31:13.798836 27238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:31:13.771686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.3:38417 (local address 127.0.0.1:60256)
0810 03:31:13.787363 (+ 15677us) negotiation.cc:107] Waiting for socket to connect
0810 03:31:13.787412 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0810 03:31:13.787632 (+ 220us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:31:13.790478 (+ 2846us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:31:13.790482 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:31:13.790514 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:31:13.790942 (+ 428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:31:13.790956 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:31:13.792810 (+ 1854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:31:13.792816 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:31:13.794528 (+ 1712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:31:13.794543 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:31:13.796556 (+ 2013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:31:13.796563 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:31:13.796857 (+ 294us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:31:13.796893 (+ 36us) client_negotiation.cc:769] Sending connection context
0810 03:31:13.797020 (+ 127us) client_negotiation.cc:240] Negotiation successful
0810 03:31:13.797227 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15337,"mutex_wait_us":73,"thread_start_us":15154,"threads_started":1}
I20260810 03:31:13.799031 27248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:31:13.773818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51052 (local address 127.27.161.1:38397)
0810 03:31:13.789534 (+ 15716us) server_negotiation.cc:206] Beginning negotiation
0810 03:31:13.789544 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 03:31:13.789585 (+ 41us) server_negotiation.cc:407] Connection header received
0810 03:31:13.789684 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:31:13.789690 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:31:13.789922 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:31:13.790147 (+ 225us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:31:13.790894 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:31:13.792420 (+ 1526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:31:13.796076 (+ 3656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:31:13.797568 (+ 1492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:31:13.797711 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:31:13.798002 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0810 03:31:13.798496 (+ 494us) server_negotiation.cc:299] Negotiation successful
0810 03:31:13.798688 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15526,"thread_start_us":15377,"threads_started":1}
I20260810 03:31:13.798915 27241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:31:13.772284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.1:38397 (local address 127.0.0.1:51052)
0810 03:31:13.788806 (+ 16522us) negotiation.cc:107] Waiting for socket to connect
0810 03:31:13.788824 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 03:31:13.788927 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:31:13.790155 (+ 1228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:31:13.790161 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:31:13.790191 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:31:13.790726 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:31:13.790745 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:31:13.792551 (+ 1806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:31:13.792556 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:31:13.795858 (+ 3302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:31:13.795873 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:31:13.798121 (+ 2248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:31:13.798128 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:31:13.798305 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:31:13.798338 (+ 33us) client_negotiation.cc:769] Sending connection context
0810 03:31:13.798423 (+ 85us) client_negotiation.cc:240] Negotiation successful
0810 03:31:13.798556 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16399,"mutex_wait_us":40,"thread_start_us":15575,"threads_started":1}
I20260810 03:31:13.802038 26907 tablet_service.cc:1467] Processing CreateTablet for tablet 41867db8e9fb43beb50ea812effd7f83 (DEFAULT_TABLE table=test-table [id=89a95eecc45c41f99bde7277643ed1e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:31:13.802707 26711 tablet_service.cc:1467] Processing CreateTablet for tablet 41867db8e9fb43beb50ea812effd7f83 (DEFAULT_TABLE table=test-table [id=89a95eecc45c41f99bde7277643ed1e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:31:13.802707 27119 tablet_service.cc:1467] Processing CreateTablet for tablet 41867db8e9fb43beb50ea812effd7f83 (DEFAULT_TABLE table=test-table [id=89a95eecc45c41f99bde7277643ed1e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:31:13.803573 26907 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41867db8e9fb43beb50ea812effd7f83. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:31:13.804168 27119 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41867db8e9fb43beb50ea812effd7f83. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:31:13.804165 26711 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41867db8e9fb43beb50ea812effd7f83. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:31:13.815802 27249 tablet_bootstrap.cc:492] T 41867db8e9fb43beb50ea812effd7f83 P 344433e16c7043da9ea03360fa7ff176: Bootstrap starting.
I20260810 03:31:13.819211 27250 tablet_bootstrap.cc:492] T 41867db8e9fb43beb50ea812effd7f83 P 9b3d9d39f2b746e4830ff9d0f3ef6457: Bootstrap starting.
I20260810 03:31:13.819569 27249 tablet_bootstrap.cc:654] T 41867db8e9fb43beb50ea812effd7f83 P 344433e16c7043da9ea03360fa7ff176: Neither blocks nor log segments found. Creating new log.
I20260810 03:31:13.820086 27251 tablet_bootstrap.cc:492] T 41867db8e9fb43beb50ea812effd7f83 P cd31b5efc51f40a4bea1e4497d2d2b4a: Bootstrap starting.
I20260810 03:31:13.821470 27249 log.cc:824] T 41867db8e9fb43beb50ea812effd7f83 P 344433e16c7043da9ea03360fa7ff176: Log is configured to *not* fsync() on all Append() calls
I20260810 03:31:13.824051 27251 tablet_bootstrap.cc:654] T 41867db8e9fb43beb50ea812effd7f83 P cd31b5efc51f40a4bea1e4497d2d2b4a: Neither blocks nor log segments found. Creating new log.
I20260810 03:31:13.825014 27250 tablet_bootstrap.cc:654] T 41867db8e9fb43beb50ea812effd7f83 P 9b3d9d39f2b746e4830ff9d0f3ef6457: Neither blocks nor log segments found. Creating new log.
I20260810 03:31:13.825681 27251 log.cc:824] T 41867db8e9fb43beb50ea812effd7f83 P cd31b5efc51f40a4bea1e4497d2d2b4a: Log is configured to *not* fsync() on all Append() calls
I20260810 03:31:13.825985 27250 log.cc:824] T 41867db8e9fb43beb50ea812effd7f83 P 9b3d9d39f2b746e4830ff9d0f3ef6457: Log is configured to *not* fsync() on all Append() calls
I20260810 03:31:13.828806 27250 tablet_bootstrap.cc:492] T 41867db8e9fb43beb50ea812effd7f83 P 9b3d9d39f2b746e4830ff9d0f3ef6457: No bootstrap required, opened a new log
I20260810 03:31:13.828926 27249 tablet_bootstrap.cc:492] T 41867db8e9fb43beb50ea812effd7f83 P 344433e16c7043da9ea03360fa7ff176: No bootstrap required, opened a new log
I20260810 03:31:13.828917 27251 tablet_bootstrap.cc:492] T 41867db8e9fb43beb50ea812effd7f83 P cd31b5efc51f40a4bea1e4497d2d2b4a: No bootstrap required, opened a new log
I20260810 03:31:13.829128 27250 ts_tablet_manager.cc:1397] T 41867db8e9fb43beb50ea812effd7f83 P 9b3d9d39f2b746e4830ff9d0f3ef6457: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.004s
I20260810 03:31:13.829221 27249 ts_tablet_manager.cc:1397] T 41867db8e9fb43beb50ea812effd7f83 P 344433e16c7043da9ea03360fa7ff176: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.005s
I20260810 03:31:13.829221 27251 ts_tablet_manager.cc:1397] T 41867db8e9fb43beb50ea812effd7f83 P cd31b5efc51f40a4bea1e4497d2d2b4a: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260810 03:31:13.837262 27250 raft_consensus.cc:348] T 41867db8e9fb43beb50ea812effd7f83 P 9b3d9d39f2b746e4830ff9d0f3ef6457 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b3d9d39f2b746e4830ff9d0f3ef6457" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 38417 } } peers { permanent_uuid: "344433e16c7043da9ea03360fa7ff176" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 41791 } } peers { permanent_uuid: "cd31b5efc51f40a4bea1e4497d2d2b4a" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 38397 } }
I20260810 03:31:13.837687 27250 raft_consensus.cc:374] T 41867db8e9fb43beb50ea812effd7f83 P 9b3d9d39f2b746e4830ff9d0f3ef6457 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:31:13.837809 27250 raft_consensus.cc:729] T 41867db8e9fb43beb50ea812effd7f83 P 9b3d9d39f2b746e4830ff9d0f3ef6457 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b3d9d39f2b746e4830ff9d0f3ef6457, State: Initialized, Role: FOLLOWER
I20260810 03:31:13.838549 27250 consensus_queue.cc:260] T 41867db8e9fb43beb50ea812effd7f83 P 9b3d9d39f2b746e4830ff9d0f3ef6457 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b3d9d39f2b746e4830ff9d0f3ef6457" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 38417 } } peers { permanent_uuid: "344433e16c7043da9ea03360fa7ff176" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 41791 } } peers { permanent_uuid: "cd31b5efc51f40a4bea1e4497d2d2b4a" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 38397 } }
I20260810 03:31:13.839812 27218 heartbeater.cc:503] Master 127.27.161.62:39261 was elected leader, sending a full tablet report...
I20260810 03:31:13.840408 27250 ts_tablet_manager.cc:1428] T 41867db8e9fb43beb50ea812effd7f83 P 9b3d9d39f2b746e4830ff9d0f3ef6457: Time spent starting tablet: real 0.011s user 0.011s sys 0.001s
I20260810 03:31:13.840130 27251 raft_consensus.cc:348] T 41867db8e9fb43beb50ea812effd7f83 P cd31b5efc51f40a4bea1e4497d2d2b4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b3d9d39f2b746e4830ff9d0f3ef6457" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 38417 } } peers { permanent_uuid: "344433e16c7043da9ea03360fa7ff176" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 41791 } } peers { permanent_uuid: "cd31b5efc51f40a4bea1e4497d2d2b4a" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 38397 } }
I20260810 03:31:13.840576 27251 raft_consensus.cc:374] T 41867db8e9fb43beb50ea812effd7f83 P cd31b5efc51f40a4bea1e4497d2d2b4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:31:13.840759 27251 raft_consensus.cc:729] T 41867db8e9fb43beb50ea812effd7f83 P cd31b5efc51f40a4bea1e4497d2d2b4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd31b5efc51f40a4bea1e4497d2d2b4a, State: Initialized, Role: FOLLOWER
I20260810 03:31:13.841075 27249 raft_consensus.cc:348] T 41867db8e9fb43beb50ea812effd7f83 P 344433e16c7043da9ea03360fa7ff176 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b3d9d39f2b746e4830ff9d0f3ef6457" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 38417 } } peers { permanent_uuid: "344433e16c7043da9ea03360fa7ff176" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 41791 } } peers { permanent_uuid: "cd31b5efc51f40a4bea1e4497d2d2b4a" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 38397 } }
I20260810 03:31:13.841475 27249 raft_consensus.cc:374] T 41867db8e9fb43beb50ea812effd7f83 P 344433e16c7043da9ea03360fa7ff176 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:31:13.841408 27251 consensus_queue.cc:260] T 41867db8e9fb43beb50ea812effd7f83 P cd31b5efc51f40a4bea1e4497d2d2b4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b3d9d39f2b746e4830ff9d0f3ef6457" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 38417 } } peers { permanent_uuid: "344433e16c7043da9ea03360fa7ff176" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 41791 } } peers { permanent_uuid: "cd31b5efc51f40a4bea1e4497d2d2b4a" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 38397 } }
I20260810 03:31:13.841598 27249 raft_consensus.cc:729] T 41867db8e9fb43beb50ea812effd7f83 P 344433e16c7043da9ea03360fa7ff176 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 344433e16c7043da9ea03360fa7ff176, State: Initialized, Role: FOLLOWER
I20260810 03:31:13.842275 27249 consensus_queue.cc:260] T 41867db8e9fb43beb50ea812effd7f83 P 344433e16c7043da9ea03360fa7ff176 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b3d9d39f2b746e4830ff9d0f3ef6457" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 38417 } } peers { permanent_uuid: "344433e16c7043da9ea03360fa7ff176" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 41791 } } peers { permanent_uuid: "cd31b5efc51f40a4bea1e4497d2d2b4a" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 38397 } }
I20260810 03:31:13.843497 27251 ts_tablet_manager.cc:1428] T 41867db8e9fb43beb50ea812effd7f83 P cd31b5efc51f40a4bea1e4497d2d2b4a: Time spent starting tablet: real 0.014s user 0.016s sys 0.000s
I20260810 03:31:13.844715 27005 heartbeater.cc:503] Master 127.27.161.62:39261 was elected leader, sending a full tablet report...
I20260810 03:31:13.845345 27249 ts_tablet_manager.cc:1428] T 41867db8e9fb43beb50ea812effd7f83 P 344433e16c7043da9ea03360fa7ff176: Time spent starting tablet: real 0.016s user 0.014s sys 0.002s
W20260810 03:31:13.962554 27220 tablet.cc:2369] T 41867db8e9fb43beb50ea812effd7f83 P 9b3d9d39f2b746e4830ff9d0f3ef6457: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 03:31:13.974284 26803 tablet.cc:2369] T 41867db8e9fb43beb50ea812effd7f83 P cd31b5efc51f40a4bea1e4497d2d2b4a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 03:31:13.978875 27008 tablet.cc:2369] T 41867db8e9fb43beb50ea812effd7f83 P 344433e16c7043da9ea03360fa7ff176: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 03:31:14.030437 27260 raft_consensus.cc:482] T 41867db8e9fb43beb50ea812effd7f83 P cd31b5efc51f40a4bea1e4497d2d2b4a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 03:31:14.030754 27260 raft_consensus.cc:504] T 41867db8e9fb43beb50ea812effd7f83 P cd31b5efc51f40a4bea1e4497d2d2b4a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b3d9d39f2b746e4830ff9d0f3ef6457" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 38417 } } peers { permanent_uuid: "344433e16c7043da9ea03360fa7ff176" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 41791 } } peers { permanent_uuid: "cd31b5efc51f40a4bea1e4497d2d2b4a" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 38397 } }
I20260810 03:31:14.032778 27260 leader_election.cc:288] T 41867db8e9fb43beb50ea812effd7f83 P cd31b5efc51f40a4bea1e4497d2d2b4a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9b3d9d39f2b746e4830ff9d0f3ef6457 (127.27.161.3:38417), 344433e16c7043da9ea03360fa7ff176 (127.27.161.2:41791)
I20260810 03:31:14.041978 27245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:31:14.033296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.161.1:53199 (local address 127.27.161.3:38417)
0810 03:31:14.033529 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0810 03:31:14.033537 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 03:31:14.034080 (+ 543us) server_negotiation.cc:407] Connection header received
0810 03:31:14.034329 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:31:14.034333 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:31:14.034410 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:31:14.034499 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:31:14.035164 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:31:14.036340 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:31:14.039436 (+ 3096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:31:14.040740 (+ 1304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:31:14.040890 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:31:14.040962 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0810 03:31:14.041554 (+ 592us) server_negotiation.cc:299] Negotiation successful
0810 03:31:14.041695 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260810 03:31:14.042155 27246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:31:14.033392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.161.1:58759 (local address 127.27.161.2:41791)
0810 03:31:14.033608 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0810 03:31:14.033614 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:31:14.034388 (+ 774us) server_negotiation.cc:407] Connection header received
0810 03:31:14.034537 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:31:14.034544 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:31:14.034639 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:31:14.034716 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:31:14.035356 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:31:14.036419 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:31:14.039466 (+ 3047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:31:14.040709 (+ 1243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:31:14.040887 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:31:14.040962 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0810 03:31:14.041554 (+ 592us) server_negotiation.cc:299] Negotiation successful
0810 03:31:14.041692 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260810 03:31:14.042284 27262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:31:14.033251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.3:38417 (local address 127.27.161.1:53199)
0810 03:31:14.033940 (+ 689us) negotiation.cc:107] Waiting for socket to connect
0810 03:31:14.033981 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0810 03:31:14.034178 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:31:14.034547 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:31:14.034551 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:31:14.034581 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:31:14.034962 (+ 381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:31:14.034984 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:31:14.036534 (+ 1550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:31:14.036540 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:31:14.039200 (+ 2660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:31:14.039216 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:31:14.041001 (+ 1785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:31:14.041011 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:31:14.041283 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:31:14.041403 (+ 120us) client_negotiation.cc:769] Sending connection context
0810 03:31:14.041524 (+ 121us) client_negotiation.cc:240] Negotiation successful
0810 03:31:14.041739 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":567,"mutex_wait_us":278,"thread_start_us":519,"threads_started":1}
I20260810 03:31:14.042243 27263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:31:14.033635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.2:41791 (local address 127.27.161.1:58759)
0810 03:31:14.034288 (+ 653us) negotiation.cc:107] Waiting for socket to connect
0810 03:31:14.034302 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 03:31:14.034419 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:31:14.034756 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:31:14.034760 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:31:14.034781 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:31:14.035176 (+ 395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:31:14.035187 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:31:14.036571 (+ 1384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:31:14.036575 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:31:14.039196 (+ 2621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:31:14.039216 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:31:14.041016 (+ 1800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:31:14.041022 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:31:14.041313 (+ 291us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:31:14.041405 (+ 92us) client_negotiation.cc:769] Sending connection context
0810 03:31:14.041524 (+ 119us) client_negotiation.cc:240] Negotiation successful
0810 03:31:14.041739 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":522,"mutex_wait_us":218,"thread_start_us":423,"threads_started":1}
I20260810 03:31:14.043483 27148 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41867db8e9fb43beb50ea812effd7f83" candidate_uuid: "cd31b5efc51f40a4bea1e4497d2d2b4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b3d9d39f2b746e4830ff9d0f3ef6457" is_pre_election: true
I20260810 03:31:14.043646 26941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41867db8e9fb43beb50ea812effd7f83" candidate_uuid: "cd31b5efc51f40a4bea1e4497d2d2b4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "344433e16c7043da9ea03360fa7ff176" is_pre_election: true
I20260810 03:31:14.044142 26941 raft_consensus.cc:2454] T 41867db8e9fb43beb50ea812effd7f83 P 344433e16c7043da9ea03360fa7ff176 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd31b5efc51f40a4bea1e4497d2d2b4a in term 0.
I20260810 03:31:14.044147 27148 raft_consensus.cc:2454] T 41867db8e9fb43beb50ea812effd7f83 P 9b3d9d39f2b746e4830ff9d0f3ef6457 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd31b5efc51f40a4bea1e4497d2d2b4a in term 0.
I20260810 03:31:14.044821 26642 leader_election.cc:302] T 41867db8e9fb43beb50ea812effd7f83 P cd31b5efc51f40a4bea1e4497d2d2b4a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 344433e16c7043da9ea03360fa7ff176, cd31b5efc51f40a4bea1e4497d2d2b4a; no voters:
I20260810 03:31:14.045253 27260 raft_consensus.cc:2781] T 41867db8e9fb43beb50ea812effd7f83 P cd31b5efc51f40a4bea1e4497d2d2b4a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 03:31:14.045400 27260 raft_consensus.cc:482] T 41867db8e9fb43beb50ea812effd7f83 P cd31b5efc51f40a4bea1e4497d2d2b4a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 03:31:14.045531 27260 raft_consensus.cc:3037] T 41867db8e9fb43beb50ea812effd7f83 P cd31b5efc51f40a4bea1e4497d2d2b4a [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:31:14.047950 27260 raft_consensus.cc:504] T 41867db8e9fb43beb50ea812effd7f83 P cd31b5efc51f40a4bea1e4497d2d2b4a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b3d9d39f2b746e4830ff9d0f3ef6457" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 38417 } } peers { permanent_uuid: "344433e16c7043da9ea03360fa7ff176" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 41791 } } peers { permanent_uuid: "cd31b5efc51f40a4bea1e4497d2d2b4a" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 38397 } }
I20260810 03:31:14.048707 27260 leader_election.cc:288] T 41867db8e9fb43beb50ea812effd7f83 P cd31b5efc51f40a4bea1e4497d2d2b4a [CANDIDATE]: Term 1 election: Requested vote from peers 9b3d9d39f2b746e4830ff9d0f3ef6457 (127.27.161.3:38417), 344433e16c7043da9ea03360fa7ff176 (127.27.161.2:41791)
I20260810 03:31:14.049177 27148 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41867db8e9fb43beb50ea812effd7f83" candidate_uuid: "cd31b5efc51f40a4bea1e4497d2d2b4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b3d9d39f2b746e4830ff9d0f3ef6457"
I20260810 03:31:14.049221 26941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41867db8e9fb43beb50ea812effd7f83" candidate_uuid: "cd31b5efc51f40a4bea1e4497d2d2b4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "344433e16c7043da9ea03360fa7ff176"
I20260810 03:31:14.049388 27148 raft_consensus.cc:3037] T 41867db8e9fb43beb50ea812effd7f83 P 9b3d9d39f2b746e4830ff9d0f3ef6457 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:31:14.049407 26941 raft_consensus.cc:3037] T 41867db8e9fb43beb50ea812effd7f83 P 344433e16c7043da9ea03360fa7ff176 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:31:14.051656 26941 raft_consensus.cc:2454] T 41867db8e9fb43beb50ea812effd7f83 P 344433e16c7043da9ea03360fa7ff176 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd31b5efc51f40a4bea1e4497d2d2b4a in term 1.
I20260810 03:31:14.052091 26642 leader_election.cc:302] T 41867db8e9fb43beb50ea812effd7f83 P cd31b5efc51f40a4bea1e4497d2d2b4a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 344433e16c7043da9ea03360fa7ff176, cd31b5efc51f40a4bea1e4497d2d2b4a; no voters:
I20260810 03:31:14.052256 27148 raft_consensus.cc:2454] T 41867db8e9fb43beb50ea812effd7f83 P 9b3d9d39f2b746e4830ff9d0f3ef6457 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd31b5efc51f40a4bea1e4497d2d2b4a in term 1.
I20260810 03:31:14.052371 27260 raft_consensus.cc:2781] T 41867db8e9fb43beb50ea812effd7f83 P cd31b5efc51f40a4bea1e4497d2d2b4a [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:31:14.052893 27260 raft_consensus.cc:686] T 41867db8e9fb43beb50ea812effd7f83 P cd31b5efc51f40a4bea1e4497d2d2b4a [term 1 LEADER]: Becoming Leader. State: Replica: cd31b5efc51f40a4bea1e4497d2d2b4a, State: Running, Role: LEADER
I20260810 03:31:14.053375 27260 consensus_queue.cc:237] T 41867db8e9fb43beb50ea812effd7f83 P cd31b5efc51f40a4bea1e4497d2d2b4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b3d9d39f2b746e4830ff9d0f3ef6457" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 38417 } } peers { permanent_uuid: "344433e16c7043da9ea03360fa7ff176" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 41791 } } peers { permanent_uuid: "cd31b5efc51f40a4bea1e4497d2d2b4a" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 38397 } }
I20260810 03:31:14.058350 26515 catalog_manager.cc:5495] T 41867db8e9fb43beb50ea812effd7f83 P cd31b5efc51f40a4bea1e4497d2d2b4a reported cstate change: term changed from 0 to 1, leader changed from <none> to cd31b5efc51f40a4bea1e4497d2d2b4a (127.27.161.1). New cstate: current_term: 1 leader_uuid: "cd31b5efc51f40a4bea1e4497d2d2b4a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b3d9d39f2b746e4830ff9d0f3ef6457" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 38417 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "344433e16c7043da9ea03360fa7ff176" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 41791 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cd31b5efc51f40a4bea1e4497d2d2b4a" member_type: VOTER last_known_addr { host: "127.27.161.1" port: 38397 } health_report { overall_health: HEALTHY } } }
I20260810 03:31:14.079778 27248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:31:14.074347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51062 (local address 127.27.161.1:38397)
0810 03:31:14.074580 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0810 03:31:14.074586 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:31:14.074607 (+ 21us) server_negotiation.cc:407] Connection header received
0810 03:31:14.074716 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:31:14.074720 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:31:14.074811 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:31:14.074951 (+ 140us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 03:31:14.075335 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:31:14.076067 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:31:14.077458 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:31:14.077885 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:31:14.078072 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:31:14.078595 (+ 523us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 03:31:14.079163 (+ 568us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 03:31:14.079321 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0810 03:31:14.079473 (+ 152us) server_negotiation.cc:299] Negotiation successful
0810 03:31:14.079567 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260810 03:31:14.088626 26941 raft_consensus.cc:1261] T 41867db8e9fb43beb50ea812effd7f83 P 344433e16c7043da9ea03360fa7ff176 [term 1 FOLLOWER]: Refusing update from remote peer cd31b5efc51f40a4bea1e4497d2d2b4a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:31:14.088626 27148 raft_consensus.cc:1261] T 41867db8e9fb43beb50ea812effd7f83 P 9b3d9d39f2b746e4830ff9d0f3ef6457 [term 1 FOLLOWER]: Refusing update from remote peer cd31b5efc51f40a4bea1e4497d2d2b4a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:31:14.089701 27267 consensus_queue.cc:1035] T 41867db8e9fb43beb50ea812effd7f83 P cd31b5efc51f40a4bea1e4497d2d2b4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "344433e16c7043da9ea03360fa7ff176" member_type: VOTER last_known_addr { host: "127.27.161.2" port: 41791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:31:14.090123 27260 consensus_queue.cc:1035] T 41867db8e9fb43beb50ea812effd7f83 P cd31b5efc51f40a4bea1e4497d2d2b4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b3d9d39f2b746e4830ff9d0f3ef6457" member_type: VOTER last_known_addr { host: "127.27.161.3" port: 38417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:31:14.103536 27276 mvcc.cc:205] Tried to move back new op lower bound from 7316818633055080448 to 7316818632926748672. Current Snapshot: MvccSnapshot[applied={T|T < 7316818633055080448}]
I20260810 03:31:14.115620 26515 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49426:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 03:31:14.115903 26515 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49426:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 03:31:14.119695 26515 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1efa32a650084dc98a1526c1eabf83ed: Sending DeleteTablet for 3 replicas of tablet 41867db8e9fb43beb50ea812effd7f83
I20260810 03:31:14.120910 27119 tablet_service.cc:1514] Processing DeleteTablet for tablet 41867db8e9fb43beb50ea812effd7f83 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:31:14 UTC) from {username='slave'} at 127.0.0.1:60256
I20260810 03:31:14.120910 26711 tablet_service.cc:1514] Processing DeleteTablet for tablet 41867db8e9fb43beb50ea812effd7f83 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:31:14 UTC) from {username='slave'} at 127.0.0.1:51052
I20260810 03:31:14.121693 26907 tablet_service.cc:1514] Processing DeleteTablet for tablet 41867db8e9fb43beb50ea812effd7f83 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:31:14 UTC) from {username='slave'} at 127.0.0.1:45800
I20260810 03:31:14.121865 28292 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iBH5/build/asan/bin/kudu pid 26616
I20260810 03:31:14.145287 27293 tablet_replica.cc:330] T 41867db8e9fb43beb50ea812effd7f83 P 344433e16c7043da9ea03360fa7ff176: stopping tablet replica
I20260810 03:31:14.146287 27293 raft_consensus.cc:2229] T 41867db8e9fb43beb50ea812effd7f83 P 344433e16c7043da9ea03360fa7ff176 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 03:31:14.147115 27293 raft_consensus.cc:2258] T 41867db8e9fb43beb50ea812effd7f83 P 344433e16c7043da9ea03360fa7ff176 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:31:14.148010 27288 tablet_replica.cc:330] T 41867db8e9fb43beb50ea812effd7f83 P 9b3d9d39f2b746e4830ff9d0f3ef6457: stopping tablet replica
I20260810 03:31:14.148537 27288 raft_consensus.cc:2229] T 41867db8e9fb43beb50ea812effd7f83 P 9b3d9d39f2b746e4830ff9d0f3ef6457 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 03:31:14.149382 27288 raft_consensus.cc:2258] T 41867db8e9fb43beb50ea812effd7f83 P 9b3d9d39f2b746e4830ff9d0f3ef6457 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:31:14.149837 27293 ts_tablet_manager.cc:1905] T 41867db8e9fb43beb50ea812effd7f83 P 344433e16c7043da9ea03360fa7ff176: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:31:14.153312 27288 ts_tablet_manager.cc:1905] T 41867db8e9fb43beb50ea812effd7f83 P 9b3d9d39f2b746e4830ff9d0f3ef6457: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:31:14.158771 27293 ts_tablet_manager.cc:1918] T 41867db8e9fb43beb50ea812effd7f83 P 344433e16c7043da9ea03360fa7ff176: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 03:31:14.159063 27293 log.cc:1196] T 41867db8e9fb43beb50ea812effd7f83 P 344433e16c7043da9ea03360fa7ff176: Deleting WAL directory at /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786332336319544-28292-0/minicluster-data/ts-1/wal/wals/41867db8e9fb43beb50ea812effd7f83
I20260810 03:31:14.159721 27293 ts_tablet_manager.cc:1939] T 41867db8e9fb43beb50ea812effd7f83 P 344433e16c7043da9ea03360fa7ff176: Deleting consensus metadata
I20260810 03:31:14.160106 27292 tablet_replica.cc:330] T 41867db8e9fb43beb50ea812effd7f83 P cd31b5efc51f40a4bea1e4497d2d2b4a: stopping tablet replica
I20260810 03:31:14.160583 27292 raft_consensus.cc:2229] T 41867db8e9fb43beb50ea812effd7f83 P cd31b5efc51f40a4bea1e4497d2d2b4a [term 1 LEADER]: Raft consensus shutting down.
I20260810 03:31:14.161278 27292 raft_consensus.cc:2258] T 41867db8e9fb43beb50ea812effd7f83 P cd31b5efc51f40a4bea1e4497d2d2b4a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:31:14.162147 26494 catalog_manager.cc:4841] TS 344433e16c7043da9ea03360fa7ff176 (127.27.161.2:41791): tablet 41867db8e9fb43beb50ea812effd7f83 (table test-table [id=89a95eecc45c41f99bde7277643ed1e9]) successfully deleted
I20260810 03:31:14.164942 27292 ts_tablet_manager.cc:1905] T 41867db8e9fb43beb50ea812effd7f83 P cd31b5efc51f40a4bea1e4497d2d2b4a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:31:14.166078 27288 ts_tablet_manager.cc:1918] T 41867db8e9fb43beb50ea812effd7f83 P 9b3d9d39f2b746e4830ff9d0f3ef6457: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 03:31:14.166199 27248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:31:14.123096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51072 (local address 127.27.161.1:38397)
0810 03:31:14.123311 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0810 03:31:14.123318 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:31:14.150759 (+ 27441us) server_negotiation.cc:407] Connection header received
0810 03:31:14.150959 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:31:14.150970 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:31:14.151120 (+ 150us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:31:14.151238 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:31:14.158294 (+ 7056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:31:14.159551 (+ 1257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:31:14.161523 (+ 1972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:31:14.162265 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:31:14.162383 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:31:14.164032 (+ 1649us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:31:14.164043 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:31:14.164072 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:31:14.164828 (+ 756us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:31:14.164884 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:31:14.164909 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:31:14.165007 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0810 03:31:14.165647 (+ 640us) server_negotiation.cc:299] Negotiation successful
0810 03:31:14.165763 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260810 03:31:14.166307 27288 log.cc:1196] T 41867db8e9fb43beb50ea812effd7f83 P 9b3d9d39f2b746e4830ff9d0f3ef6457: Deleting WAL directory at /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786332336319544-28292-0/minicluster-data/ts-2/wal/wals/41867db8e9fb43beb50ea812effd7f83
I20260810 03:31:14.166865 27288 ts_tablet_manager.cc:1939] T 41867db8e9fb43beb50ea812effd7f83 P 9b3d9d39f2b746e4830ff9d0f3ef6457: Deleting consensus metadata
I20260810 03:31:14.167034 26797 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:31:14.168738 26491 catalog_manager.cc:4841] TS 9b3d9d39f2b746e4830ff9d0f3ef6457 (127.27.161.3:38417): tablet 41867db8e9fb43beb50ea812effd7f83 (table test-table [id=89a95eecc45c41f99bde7277643ed1e9]) successfully deleted
I20260810 03:31:14.275988 28292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iBH5/build/asan/bin/kudu with pid 26616
I20260810 03:31:14.300704 28292 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iBH5/build/asan/bin/kudu pid 26807
W20260810 03:31:14.300670 26492 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.161.1:38397 (error 108)
I20260810 03:31:14.302642 27297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:31:14.301853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.1:38397 (local address 127.0.0.1:51074)
0810 03:31:14.302316 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0810 03:31:14.302460 (+ 144us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.1:38397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":240,"threads_started":1}
W20260810 03:31:14.302908 26492 catalog_manager.cc:4584] TS cd31b5efc51f40a4bea1e4497d2d2b4a (127.27.161.1:38397): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 41867db8e9fb43beb50ea812effd7f83: Network error: Client connection negotiation failed: client connection to 127.27.161.1:38397: connect: Connection refused (error 111)
I20260810 03:31:14.307830 27246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:31:14.301451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45810 (local address 127.27.161.2:41791)
0810 03:31:14.301649 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0810 03:31:14.301655 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:31:14.301736 (+ 81us) server_negotiation.cc:407] Connection header received
0810 03:31:14.301917 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:31:14.301921 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:31:14.302049 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:31:14.302159 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:31:14.302659 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:31:14.303507 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:31:14.304835 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:31:14.305231 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:31:14.305317 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:31:14.306449 (+ 1132us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:31:14.306459 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:31:14.306474 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:31:14.307073 (+ 599us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:31:14.307126 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:31:14.307146 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:31:14.307232 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0810 03:31:14.307379 (+ 147us) server_negotiation.cc:299] Negotiation successful
0810 03:31:14.307473 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260810 03:31:14.308542 26999 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:31:14.356453 27297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:31:14.355923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.1:38397 (local address 127.0.0.1:51080)
0810 03:31:14.356160 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0810 03:31:14.356275 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.1:38397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260810 03:31:14.357533 27297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:31:14.357133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.1:38397 (local address 127.0.0.1:51092)
0810 03:31:14.357283 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0810 03:31:14.357369 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.1:38397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260810 03:31:14.397644 28292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iBH5/build/asan/bin/kudu with pid 26807
I20260810 03:31:14.418393 27297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:31:14.417794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.1:38397 (local address 127.0.0.1:51096)
0810 03:31:14.418071 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0810 03:31:14.418192 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.1:38397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260810 03:31:14.419624 27297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:31:14.419184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.1:38397 (local address 127.0.0.1:51100)
0810 03:31:14.419346 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0810 03:31:14.419435 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.1:38397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260810 03:31:14.425089 28292 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iBH5/build/asan/bin/kudu pid 27011
I20260810 03:31:14.431887 27245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:31:14.425916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60268 (local address 127.27.161.3:38417)
0810 03:31:14.426280 (+ 364us) server_negotiation.cc:206] Beginning negotiation
0810 03:31:14.426285 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:31:14.426304 (+ 19us) server_negotiation.cc:407] Connection header received
0810 03:31:14.426362 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:31:14.426366 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:31:14.426456 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:31:14.426547 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:31:14.426899 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:31:14.427629 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:31:14.428948 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:31:14.429372 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:31:14.429478 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:31:14.430538 (+ 1060us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:31:14.430547 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:31:14.430565 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:31:14.431176 (+ 611us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:31:14.431228 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:31:14.431252 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:31:14.431345 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0810 03:31:14.431509 (+ 164us) server_negotiation.cc:299] Negotiation successful
0810 03:31:14.431616 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":247}
I20260810 03:31:14.432585 27209 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:31:14.500371 27297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:31:14.499884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.1:38397 (local address 127.0.0.1:51114)
0810 03:31:14.500109 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0810 03:31:14.500217 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.1:38397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260810 03:31:14.501596 27297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:31:14.501057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.1:38397 (local address 127.0.0.1:51130)
0810 03:31:14.501271 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0810 03:31:14.501343 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.1:38397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112}
I20260810 03:31:14.522317 28292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iBH5/build/asan/bin/kudu with pid 27011
I20260810 03:31:14.544608 28292 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iBH5/build/asan/bin/kudu pid 26472
I20260810 03:31:14.545691 26571 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:31:14.654714 28292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iBH5/build/asan/bin/kudu with pid 26472
2026-08-10T03:31:14Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (3092 ms)
[----------] 6 tests from EncryptionPolicyTest (23864 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786332336319544-28292-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 03:31:14 dist-test-slave-vpvm krb5kdc[27303](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 03:31:14 dist-test-slave-vpvm krb5kdc[27303](info): set up 2 sockets
Aug 10 03:31:14 dist-test-slave-vpvm krb5kdc[27303](info): commencing operation
krb5kdc: starting...
W20260810 03:31:15.931151 28292 mini_kdc.cc:121] Time spent starting KDC: real 1.221s user 0.006s sys 0.049s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786332336319544-28292-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786332336319544-28292-0/krb5kdc/test-user.keytab.
Aug 10 03:31:16 dist-test-slave-vpvm krb5kdc[27303](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332676, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T03:31:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T03:31:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786332336319544-28292-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786332336319544-28292-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 03:31:16.291723 28292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iBH5/build/asan/bin/kudu
/tmp/dist-test-task24iBH5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786332336319544-28292-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786332336319544-28292-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786332336319544-28292-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786332336319544-28292-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44695
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786332336319544-28292-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44695
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786332336319544-28292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786332336319544-28292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786332336319544-28292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:31:16.650533 27322 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:31:16.650864 27322 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:31:16.651008 27322 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:31:16.658543 27322 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:31:16.658694 27322 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:31:16.658775 27322 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:31:16.658862 27322 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:31:16.658924 27322 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:31:16.669236 27322 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786332336319544-28292-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786332336319544-28292-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44695
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786332336319544-28292-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44695
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786332336319544-28292-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786332336319544-28292-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:31:16.670650 27322 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:31:16.671980 27322 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:31:16.684001 27338 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:31:16.684088 27332 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:31:16.684274 27331 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:31:16.685163 27322 server_base.cc:1034] running on GCE node
I20260810 03:31:16.686031 27322 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:31:16.687527 27322 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:31:16.688758 27322 hybrid_clock.cc:648] HybridClock initialized: now 1786332676688699 us; error 50 us; skew 500 ppm
I20260810 03:31:16.689136 27322 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:31:16 dist-test-slave-vpvm krb5kdc[27303](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332676, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:31:16.696658 27322 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 03:31:16.698426 27322 webserver.cc:460] Webserver started at http://127.0.0.1:38685/ using document root <none> and password file <none>
I20260810 03:31:16.699167 27322 fs_manager.cc:359] Metadata directory not provided
I20260810 03:31:16.699299 27322 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:31:16.699620 27322 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:31:16.703055 27322 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786332336319544-28292-0/minicluster-data/master-0/data/instance:
uuid: "634806ce11d04856bd656abf15fc6d92"
format_stamp: "Formatted at 2026-08-10 03:31:16 on dist-test-slave-vpvm"
server_key: "2c6fe67dd233ee22e2b96380c7e27070"
server_key_iv: "11bd4676b6943ae8a6a580bf75c268b3"
server_key_version: "encryptionkey@0"
I20260810 03:31:16.703924 27322 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786332336319544-28292-0/minicluster-data/master-0/wal/instance:
uuid: "634806ce11d04856bd656abf15fc6d92"
format_stamp: "Formatted at 2026-08-10 03:31:16 on dist-test-slave-vpvm"
server_key: "2c6fe67dd233ee22e2b96380c7e27070"
server_key_iv: "11bd4676b6943ae8a6a580bf75c268b3"
server_key_version: "encryptionkey@0"
I20260810 03:31:16.709566 27322 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260810 03:31:16.713271 27348 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:31:16.714680 27322 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260810 03:31:16.714866 27322 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786332336319544-28292-0/minicluster-data/master-0/data,/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786332336319544-28292-0/minicluster-data/master-0/wal
uuid: "634806ce11d04856bd656abf15fc6d92"
format_stamp: "Formatted at 2026-08-10 03:31:16 on dist-test-slave-vpvm"
server_key: "2c6fe67dd233ee22e2b96380c7e27070"
server_key_iv: "11bd4676b6943ae8a6a580bf75c268b3"
server_key_version: "encryptionkey@0"
I20260810 03:31:16.715075 27322 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786332336319544-28292-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786332336319544-28292-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786332336319544-28292-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:31:16.736596 27322 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:31:16.740972 27322 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:31:16.741277 27322 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:31:16.759608 27322 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44695
I20260810 03:31:16.759616 27425 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44695 every 8 connection(s)
I20260810 03:31:16.761096 27322 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786332336319544-28292-0/minicluster-data/master-0/data/info.pb
I20260810 03:31:16.764976 27426 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:31:16.766782 28292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iBH5/build/asan/bin/kudu as pid 27322
I20260810 03:31:16.766981 28292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786332336319544-28292-0/minicluster-data/master-0/wal/instance
I20260810 03:31:16.767596 28292 external_mini_cluster.cc:1428] Setting key 0645cc57f819c408c89349aaedc85a5a
I20260810 03:31:16.778791 27426 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 634806ce11d04856bd656abf15fc6d92: Bootstrap starting.
Aug 10 03:31:16 dist-test-slave-vpvm krb5kdc[27303](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332676, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260810 03:31:16.782436 27426 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 634806ce11d04856bd656abf15fc6d92: Neither blocks nor log segments found. Creating new log.
I20260810 03:31:16.783787 27426 log.cc:824] T 00000000000000000000000000000000 P 634806ce11d04856bd656abf15fc6d92: Log is configured to *not* fsync() on all Append() calls
I20260810 03:31:16.787437 27426 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 634806ce11d04856bd656abf15fc6d92: No bootstrap required, opened a new log
I20260810 03:31:16.791834 27429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:31:16.769683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59703 (local address 127.0.0.1:44695)
0810 03:31:16.770593 (+ 910us) server_negotiation.cc:206] Beginning negotiation
0810 03:31:16.770606 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 03:31:16.770653 (+ 47us) server_negotiation.cc:407] Connection header received
0810 03:31:16.772358 (+ 1705us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:31:16.772389 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:31:16.772875 (+ 486us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:31:16.773296 (+ 421us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:31:16.774949 (+ 1653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:31:16.776344 (+ 1395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:31:16.777691 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:31:16.778444 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:31:16.778613 (+ 169us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:31:16.783277 (+ 4664us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:31:16.783290 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:31:16.783311 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:31:16.783365 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:31:16.786945 (+ 3580us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:31:16.788014 (+ 1069us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:31:16.788022 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:31:16.788032 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:31:16.788233 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:31:16.788777 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:31:16.788781 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:31:16.788783 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:31:16.789370 (+ 587us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:31:16.789646 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0810 03:31:16.790288 (+ 642us) server_negotiation.cc:299] Negotiation successful
0810 03:31:16.790673 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":655,"thread_start_us":399,"threads_started":1}
I20260810 03:31:16.796317 28292 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 03:31:16.796929 27426 raft_consensus.cc:348] T 00000000000000000000000000000000 P 634806ce11d04856bd656abf15fc6d92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "634806ce11d04856bd656abf15fc6d92" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44695 } }
I20260810 03:31:16.797392 27426 raft_consensus.cc:374] T 00000000000000000000000000000000 P 634806ce11d04856bd656abf15fc6d92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:31:16.797487 27426 raft_consensus.cc:729] T 00000000000000000000000000000000 P 634806ce11d04856bd656abf15fc6d92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 634806ce11d04856bd656abf15fc6d92, State: Initialized, Role: FOLLOWER
I20260810 03:31:16.798193 27426 consensus_queue.cc:260] T 00000000000000000000000000000000 P 634806ce11d04856bd656abf15fc6d92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "634806ce11d04856bd656abf15fc6d92" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44695 } }
I20260810 03:31:16.798484 27426 raft_consensus.cc:388] T 00000000000000000000000000000000 P 634806ce11d04856bd656abf15fc6d92 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 03:31:16.798630 27426 raft_consensus.cc:482] T 00000000000000000000000000000000 P 634806ce11d04856bd656abf15fc6d92 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 03:31:16.798816 27426 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 634806ce11d04856bd656abf15fc6d92 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:31:16.802232 27426 raft_consensus.cc:504] T 00000000000000000000000000000000 P 634806ce11d04856bd656abf15fc6d92 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "634806ce11d04856bd656abf15fc6d92" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44695 } }
I20260810 03:31:16.802839 27426 leader_election.cc:302] T 00000000000000000000000000000000 P 634806ce11d04856bd656abf15fc6d92 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 634806ce11d04856bd656abf15fc6d92; no voters:
I20260810 03:31:16.803512 27426 leader_election.cc:288] T 00000000000000000000000000000000 P 634806ce11d04856bd656abf15fc6d92 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 03:31:16.803627 27438 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 634806ce11d04856bd656abf15fc6d92 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:31:16.805838 27438 raft_consensus.cc:686] T 00000000000000000000000000000000 P 634806ce11d04856bd656abf15fc6d92 [term 1 LEADER]: Becoming Leader. State: Replica: 634806ce11d04856bd656abf15fc6d92, State: Running, Role: LEADER
I20260810 03:31:16.806110 27426 sys_catalog.cc:564] T 00000000000000000000000000000000 P 634806ce11d04856bd656abf15fc6d92 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:31:16.806571 27438 consensus_queue.cc:237] T 00000000000000000000000000000000 P 634806ce11d04856bd656abf15fc6d92 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "634806ce11d04856bd656abf15fc6d92" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44695 } }
I20260810 03:31:16.815181 27439 sys_catalog.cc:455] T 00000000000000000000000000000000 P 634806ce11d04856bd656abf15fc6d92 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "634806ce11d04856bd656abf15fc6d92" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "634806ce11d04856bd656abf15fc6d92" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44695 } } }
I20260810 03:31:16.815709 27439 sys_catalog.cc:458] T 00000000000000000000000000000000 P 634806ce11d04856bd656abf15fc6d92 [sys.catalog]: This master's current role is: LEADER
I20260810 03:31:16.819813 27429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:31:16.802430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41791 (local address 127.0.0.1:44695)
0810 03:31:16.802603 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0810 03:31:16.802609 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:31:16.803295 (+ 686us) server_negotiation.cc:407] Connection header received
0810 03:31:16.803483 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:31:16.803487 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:31:16.803563 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:31:16.803657 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:31:16.805597 (+ 1940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:31:16.806667 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:31:16.808301 (+ 1634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:31:16.808807 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:31:16.808922 (+ 115us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:31:16.811542 (+ 2620us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:31:16.811547 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:31:16.811550 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:31:16.811587 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:31:16.814749 (+ 3162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:31:16.815955 (+ 1206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:31:16.815961 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:31:16.815964 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:31:16.816140 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:31:16.816792 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:31:16.816798 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:31:16.816802 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:31:16.817327 (+ 525us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:31:16.817641 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0810 03:31:16.818949 (+ 1308us) server_negotiation.cc:299] Negotiation successful
0810 03:31:16.819182 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260810 03:31:16.824419 27442 sys_catalog.cc:455] T 00000000000000000000000000000000 P 634806ce11d04856bd656abf15fc6d92 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 634806ce11d04856bd656abf15fc6d92. Latest consensus state: current_term: 1 leader_uuid: "634806ce11d04856bd656abf15fc6d92" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "634806ce11d04856bd656abf15fc6d92" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44695 } } }
I20260810 03:31:16.824728 27442 sys_catalog.cc:458] T 00000000000000000000000000000000 P 634806ce11d04856bd656abf15fc6d92 [sys.catalog]: This master's current role is: LEADER
I20260810 03:31:16.825296 27455 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 03:31:16.831182 27455 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 03:31:16.840803 27455 catalog_manager.cc:1347] Generated new cluster ID: a8eaa5aa75384e579c42c18848462207
I20260810 03:31:16.840945 27455 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 03:31:16.851711 27455 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 03:31:16.853081 27455 catalog_manager.cc:1504] Loading token signing keys...
I20260810 03:31:16.862625 27455 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 634806ce11d04856bd656abf15fc6d92: Generated new TSK 0
I20260810 03:31:16.863546 27455 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 03:31:16.870683 28292 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iBH5/build/asan/bin/kudu pid 27322
I20260810 03:31:16.872063 27421 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:31:16.962949 28292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iBH5/build/asan/bin/kudu with pid 27322
W20260810 03:31:16.988073 27301 connection.cc:535] client connection to 127.0.0.1:44695 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44695): Connection reset by peer (error 104)
2026-08-10T03:31:16Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (2297 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786332336319544-28292-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 03:31:17 dist-test-slave-vpvm krb5kdc[27478](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 03:31:17 dist-test-slave-vpvm krb5kdc[27478](info): set up 2 sockets
Aug 10 03:31:17 dist-test-slave-vpvm krb5kdc[27478](info): commencing operation
krb5kdc: starting...
W20260810 03:31:18.228322 28292 mini_kdc.cc:121] Time spent starting KDC: real 1.220s user 0.000s sys 0.060s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786332336319544-28292-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786332336319544-28292-0/krb5kdc/test-user.keytab.
Aug 10 03:31:18 dist-test-slave-vpvm krb5kdc[27478](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332678, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T03:31:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T03:31:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786332336319544-28292-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786332336319544-28292-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 03:31:18.595227 28292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iBH5/build/asan/bin/kudu
/tmp/dist-test-task24iBH5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786332336319544-28292-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786332336319544-28292-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786332336319544-28292-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786332336319544-28292-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38423
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786332336319544-28292-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38423
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786332336319544-28292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786332336319544-28292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786332336319544-28292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:31:18.950234 27495 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:31:18.950562 27495 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:31:18.950659 27495 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:31:18.958094 27495 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:31:18.958222 27495 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:31:18.958280 27495 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:31:18.958340 27495 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:31:18.958384 27495 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:31:18.969355 27495 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786332336319544-28292-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786332336319544-28292-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38423
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786332336319544-28292-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38423
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786332336319544-28292-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786332336319544-28292-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:31:18.971045 27495 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:31:18.972404 27495 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:31:18.984982 27502 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:31:18.984949 27503 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:31:18.985031 27505 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:31:18.985852 27495 server_base.cc:1034] running on GCE node
I20260810 03:31:18.986732 27495 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:31:18.988296 27495 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:31:18.989564 27495 hybrid_clock.cc:648] HybridClock initialized: now 1786332678989488 us; error 63 us; skew 500 ppm
I20260810 03:31:18.989943 27495 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:31:18 dist-test-slave-vpvm krb5kdc[27478](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332678, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:31:18.993232 27495 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 03:31:18.994874 27495 webserver.cc:460] Webserver started at http://127.0.0.1:35927/ using document root <none> and password file <none>
I20260810 03:31:18.995573 27495 fs_manager.cc:359] Metadata directory not provided
I20260810 03:31:18.995695 27495 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:31:18.995998 27495 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:31:18.999138 27495 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786332336319544-28292-0/minicluster-data/master-0/data/instance:
uuid: "706f9af33b764c18b99bf2f3a5ea408d"
format_stamp: "Formatted at 2026-08-10 03:31:18 on dist-test-slave-vpvm"
server_key: "da8ce36b57681daa6bdde1e3ce5c3e18"
server_key_iv: "86c8f1d4cca2e312cba7f67d02f2ffcf"
server_key_version: "encryptionkey@0"
I20260810 03:31:18.999918 27495 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786332336319544-28292-0/minicluster-data/master-0/wal/instance:
uuid: "706f9af33b764c18b99bf2f3a5ea408d"
format_stamp: "Formatted at 2026-08-10 03:31:18 on dist-test-slave-vpvm"
server_key: "da8ce36b57681daa6bdde1e3ce5c3e18"
server_key_iv: "86c8f1d4cca2e312cba7f67d02f2ffcf"
server_key_version: "encryptionkey@0"
I20260810 03:31:19.005173 27495 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260810 03:31:19.008826 27516 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:31:19.010288 27495 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 03:31:19.010469 27495 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786332336319544-28292-0/minicluster-data/master-0/data,/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786332336319544-28292-0/minicluster-data/master-0/wal
uuid: "706f9af33b764c18b99bf2f3a5ea408d"
format_stamp: "Formatted at 2026-08-10 03:31:18 on dist-test-slave-vpvm"
server_key: "da8ce36b57681daa6bdde1e3ce5c3e18"
server_key_iv: "86c8f1d4cca2e312cba7f67d02f2ffcf"
server_key_version: "encryptionkey@0"
I20260810 03:31:19.010674 27495 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786332336319544-28292-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786332336319544-28292-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786332336319544-28292-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:31:19.037551 27495 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:31:19.042162 27495 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:31:19.042461 27495 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:31:19.060302 27495 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38423
I20260810 03:31:19.060333 27595 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38423 every 8 connection(s)
I20260810 03:31:19.061805 27495 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786332336319544-28292-0/minicluster-data/master-0/data/info.pb
I20260810 03:31:19.065985 27597 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:31:19.070433 28292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iBH5/build/asan/bin/kudu as pid 27495
I20260810 03:31:19.070622 28292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786332336319544-28292-0/minicluster-data/master-0/wal/instance
I20260810 03:31:19.071192 28292 external_mini_cluster.cc:1428] Setting key f0a6c9417d42378041f7cbc9e4761432
I20260810 03:31:19.077978 27597 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 706f9af33b764c18b99bf2f3a5ea408d: Bootstrap starting.
I20260810 03:31:19.081319 27597 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 706f9af33b764c18b99bf2f3a5ea408d: Neither blocks nor log segments found. Creating new log.
I20260810 03:31:19.082557 27597 log.cc:824] T 00000000000000000000000000000000 P 706f9af33b764c18b99bf2f3a5ea408d: Log is configured to *not* fsync() on all Append() calls
I20260810 03:31:19.086084 27603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:31:19.073332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44561 (local address 127.0.0.1:38423)
0810 03:31:19.074253 (+ 921us) server_negotiation.cc:206] Beginning negotiation
0810 03:31:19.074266 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 03:31:19.074313 (+ 47us) server_negotiation.cc:407] Connection header received
0810 03:31:19.076491 (+ 2178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:31:19.076521 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:31:19.076994 (+ 473us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:31:19.077390 (+ 396us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:31:19.078002 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:31:19.079390 (+ 1388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:31:19.080771 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:31:19.081526 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:31:19.081732 (+ 206us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:31:19.083112 (+ 1380us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:31:19.083127 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:31:19.083151 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:31:19.084042 (+ 891us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:31:19.084155 (+ 113us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:31:19.084191 (+ 36us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:31:19.084355 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0810 03:31:19.084821 (+ 466us) server_negotiation.cc:299] Negotiation successful
0810 03:31:19.085018 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":658,"thread_start_us":380,"threads_started":1}
I20260810 03:31:19.086300 27597 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 706f9af33b764c18b99bf2f3a5ea408d: No bootstrap required, opened a new log
I20260810 03:31:19.090265 28292 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 03:31:19.094978 27597 raft_consensus.cc:348] T 00000000000000000000000000000000 P 706f9af33b764c18b99bf2f3a5ea408d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "706f9af33b764c18b99bf2f3a5ea408d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38423 } }
I20260810 03:31:19.095530 27597 raft_consensus.cc:374] T 00000000000000000000000000000000 P 706f9af33b764c18b99bf2f3a5ea408d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:31:19.095652 27597 raft_consensus.cc:729] T 00000000000000000000000000000000 P 706f9af33b764c18b99bf2f3a5ea408d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 706f9af33b764c18b99bf2f3a5ea408d, State: Initialized, Role: FOLLOWER
I20260810 03:31:19.096418 27597 consensus_queue.cc:260] T 00000000000000000000000000000000 P 706f9af33b764c18b99bf2f3a5ea408d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "706f9af33b764c18b99bf2f3a5ea408d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38423 } }
I20260810 03:31:19.096705 27597 raft_consensus.cc:388] T 00000000000000000000000000000000 P 706f9af33b764c18b99bf2f3a5ea408d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 03:31:19.096880 27597 raft_consensus.cc:482] T 00000000000000000000000000000000 P 706f9af33b764c18b99bf2f3a5ea408d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 03:31:19.097095 27597 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 706f9af33b764c18b99bf2f3a5ea408d [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:31:19.101111 27597 raft_consensus.cc:504] T 00000000000000000000000000000000 P 706f9af33b764c18b99bf2f3a5ea408d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "706f9af33b764c18b99bf2f3a5ea408d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38423 } }
I20260810 03:31:19.101805 27597 leader_election.cc:302] T 00000000000000000000000000000000 P 706f9af33b764c18b99bf2f3a5ea408d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 706f9af33b764c18b99bf2f3a5ea408d; no voters:
I20260810 03:31:19.102596 27603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:31:19.095208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59783 (local address 127.0.0.1:38423)
0810 03:31:19.095419 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0810 03:31:19.095426 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:31:19.095734 (+ 308us) server_negotiation.cc:407] Connection header received
0810 03:31:19.095941 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:31:19.095947 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:31:19.096038 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:31:19.096175 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:31:19.096700 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:31:19.097785 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:31:19.099420 (+ 1635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:31:19.100118 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:31:19.100269 (+ 151us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:31:19.101636 (+ 1367us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:31:19.101643 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:31:19.101648 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:31:19.101675 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:31:19.101730 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:31:19.101752 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:31:19.101889 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0810 03:31:19.102179 (+ 290us) server_negotiation.cc:299] Negotiation successful
0810 03:31:19.102242 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260810 03:31:19.102655 27597 leader_election.cc:288] T 00000000000000000000000000000000 P 706f9af33b764c18b99bf2f3a5ea408d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 03:31:19.102780 27611 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 706f9af33b764c18b99bf2f3a5ea408d [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:31:19.105154 27597 sys_catalog.cc:564] T 00000000000000000000000000000000 P 706f9af33b764c18b99bf2f3a5ea408d [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:31:19.106292 27611 raft_consensus.cc:686] T 00000000000000000000000000000000 P 706f9af33b764c18b99bf2f3a5ea408d [term 1 LEADER]: Becoming Leader. State: Replica: 706f9af33b764c18b99bf2f3a5ea408d, State: Running, Role: LEADER
I20260810 03:31:19.106984 27611 consensus_queue.cc:237] T 00000000000000000000000000000000 P 706f9af33b764c18b99bf2f3a5ea408d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "706f9af33b764c18b99bf2f3a5ea408d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38423 } }
I20260810 03:31:19.110298 27613 sys_catalog.cc:455] T 00000000000000000000000000000000 P 706f9af33b764c18b99bf2f3a5ea408d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 706f9af33b764c18b99bf2f3a5ea408d. Latest consensus state: current_term: 1 leader_uuid: "706f9af33b764c18b99bf2f3a5ea408d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "706f9af33b764c18b99bf2f3a5ea408d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38423 } } }
I20260810 03:31:19.110759 27613 sys_catalog.cc:458] T 00000000000000000000000000000000 P 706f9af33b764c18b99bf2f3a5ea408d [sys.catalog]: This master's current role is: LEADER
I20260810 03:31:19.110863 27612 sys_catalog.cc:455] T 00000000000000000000000000000000 P 706f9af33b764c18b99bf2f3a5ea408d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "706f9af33b764c18b99bf2f3a5ea408d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "706f9af33b764c18b99bf2f3a5ea408d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38423 } } }
I20260810 03:31:19.111164 27612 sys_catalog.cc:458] T 00000000000000000000000000000000 P 706f9af33b764c18b99bf2f3a5ea408d [sys.catalog]: This master's current role is: LEADER
I20260810 03:31:19.116513 27627 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 03:31:19.121162 27627 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 03:31:19.130405 27627 catalog_manager.cc:1347] Generated new cluster ID: 1e5a6d0f081d4a708545b358de6df7da
I20260810 03:31:19.130553 27627 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 03:31:19.152832 27627 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 03:31:19.154495 27627 catalog_manager.cc:1504] Loading token signing keys...
I20260810 03:31:19.168511 27627 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 706f9af33b764c18b99bf2f3a5ea408d: Generated new TSK 0
I20260810 03:31:19.169746 27627 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 03:31:19.183439 28292 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iBH5/build/asan/bin/kudu pid 27495
W20260810 03:31:19.184790 27589 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:44561
W20260810 03:31:19.185672 28292 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task24iBH5/build/asan/bin/kudu pid 27495: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260810 03:31:19.185851 28292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iBH5/build/asan/bin/kudu with pid 27495
W20260810 03:31:19.208216 27475 connection.cc:535] client connection to 127.0.0.1:38423 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38423): Connection reset by peer (error 104)
2026-08-10T03:31:19Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (2220 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786332336319544-28292-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 03:31:19 dist-test-slave-vpvm krb5kdc[27643](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 03:31:19 dist-test-slave-vpvm krb5kdc[27643](info): set up 2 sockets
Aug 10 03:31:19 dist-test-slave-vpvm krb5kdc[27643](info): commencing operation
krb5kdc: starting...
W20260810 03:31:20.464548 28292 mini_kdc.cc:121] Time spent starting KDC: real 1.236s user 0.000s sys 0.058s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786332336319544-28292-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786332336319544-28292-0/krb5kdc/test-user.keytab.
Aug 10 03:31:20 dist-test-slave-vpvm krb5kdc[27643](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332680, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T03:31:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T03:31:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786332336319544-28292-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786332336319544-28292-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 03:31:20.818189 28292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iBH5/build/asan/bin/kudu
/tmp/dist-test-task24iBH5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786332336319544-28292-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786332336319544-28292-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786332336319544-28292-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786332336319544-28292-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39951
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786332336319544-28292-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39081
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39951
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786332336319544-28292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786332336319544-28292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786332336319544-28292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:31:21.180272 27663 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:31:21.180583 27663 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:31:21.180720 27663 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:31:21.188038 27663 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:31:21.188161 27663 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:31:21.188218 27663 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:31:21.188306 27663 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:31:21.188359 27663 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:31:21.199504 27663 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39081
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786332336319544-28292-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786332336319544-28292-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39951
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786332336319544-28292-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39951
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786332336319544-28292-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786332336319544-28292-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:31:21.201031 27663 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:31:21.202476 27663 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:31:21.215304 27675 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:31:21.215523 27671 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:31:21.215348 27677 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:31:21.216639 27663 server_base.cc:1034] running on GCE node
I20260810 03:31:21.217525 27663 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:31:21.219226 27663 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:31:21.220482 27663 hybrid_clock.cc:648] HybridClock initialized: now 1786332681220422 us; error 48 us; skew 500 ppm
I20260810 03:31:21.220884 27663 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:31:21 dist-test-slave-vpvm krb5kdc[27643](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332681, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:31:21.224277 27663 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 03:31:21.225929 27663 webserver.cc:460] Webserver started at http://127.0.0.1:42395/ using document root <none> and password file <none>
I20260810 03:31:21.226689 27663 fs_manager.cc:359] Metadata directory not provided
I20260810 03:31:21.226823 27663 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:31:21.227145 27663 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:31:21.230566 27663 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786332336319544-28292-0/minicluster-data/master-0/data/instance:
uuid: "25668d9bb9f143e3bd9f7e10279bcf73"
format_stamp: "Formatted at 2026-08-10 03:31:21 on dist-test-slave-vpvm"
server_key: "e3158ed304f63e34024b1832697e1465"
server_key_iv: "cdd22a72e2390ac1190422748ec7dd18"
server_key_version: "encryptionkey@0"
I20260810 03:31:21.231389 27663 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786332336319544-28292-0/minicluster-data/master-0/wal/instance:
uuid: "25668d9bb9f143e3bd9f7e10279bcf73"
format_stamp: "Formatted at 2026-08-10 03:31:21 on dist-test-slave-vpvm"
server_key: "e3158ed304f63e34024b1832697e1465"
server_key_iv: "cdd22a72e2390ac1190422748ec7dd18"
server_key_version: "encryptionkey@0"
I20260810 03:31:21.237108 27663 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260810 03:31:21.241285 27686 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:31:21.242889 27663 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 03:31:21.243083 27663 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786332336319544-28292-0/minicluster-data/master-0/data,/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786332336319544-28292-0/minicluster-data/master-0/wal
uuid: "25668d9bb9f143e3bd9f7e10279bcf73"
format_stamp: "Formatted at 2026-08-10 03:31:21 on dist-test-slave-vpvm"
server_key: "e3158ed304f63e34024b1832697e1465"
server_key_iv: "cdd22a72e2390ac1190422748ec7dd18"
server_key_version: "encryptionkey@0"
I20260810 03:31:21.243319 27663 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786332336319544-28292-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786332336319544-28292-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786332336319544-28292-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:31:21.249673 27663 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:31:21.255084 27663 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:31:21.255399 27663 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:31:21.273945 27663 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39951
I20260810 03:31:21.274020 27755 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39951 every 8 connection(s)
I20260810 03:31:21.275506 27663 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786332336319544-28292-0/minicluster-data/master-0/data/info.pb
I20260810 03:31:21.279465 27756 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:31:21.285006 28292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iBH5/build/asan/bin/kudu as pid 27663
I20260810 03:31:21.285189 28292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786332336319544-28292-0/minicluster-data/master-0/wal/instance
I20260810 03:31:21.285734 28292 external_mini_cluster.cc:1428] Setting key c93fa4f92edc141e2861321843543e4f
I20260810 03:31:21.291668 27756 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 25668d9bb9f143e3bd9f7e10279bcf73: Bootstrap starting.
I20260810 03:31:21.294467 27761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:31:21.287710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56923 (local address 127.0.0.1:39951)
0810 03:31:21.288657 (+ 947us) server_negotiation.cc:206] Beginning negotiation
0810 03:31:21.288673 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 03:31:21.288720 (+ 47us) server_negotiation.cc:407] Connection header received
0810 03:31:21.290763 (+ 2043us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:31:21.290795 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:31:21.291220 (+ 425us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:31:21.291608 (+ 388us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:31:21.292780 (+ 1172us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:31:21.292787 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:31:21.292802 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:31:21.292817 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:31:21.292884 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:31:21.292905 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:31:21.293080 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0810 03:31:21.293189 (+ 109us) server_negotiation.cc:299] Negotiation successful
0810 03:31:21.293384 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":692,"thread_start_us":404,"threads_started":1}
I20260810 03:31:21.295502 27756 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 25668d9bb9f143e3bd9f7e10279bcf73: Neither blocks nor log segments found. Creating new log.
I20260810 03:31:21.296640 27756 log.cc:824] T 00000000000000000000000000000000 P 25668d9bb9f143e3bd9f7e10279bcf73: Log is configured to *not* fsync() on all Append() calls
I20260810 03:31:21.298476 28292 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 03:31:21.299791 27756 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 25668d9bb9f143e3bd9f7e10279bcf73: No bootstrap required, opened a new log
I20260810 03:31:21.305528 27761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:31:21.303030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55013 (local address 127.0.0.1:39951)
0810 03:31:21.303261 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0810 03:31:21.303272 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 03:31:21.303602 (+ 330us) server_negotiation.cc:407] Connection header received
0810 03:31:21.303785 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:31:21.303789 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:31:21.303840 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:31:21.303935 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:31:21.304885 (+ 950us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:31:21.304893 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:31:21.304897 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:31:21.304914 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:31:21.304958 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:31:21.304973 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:31:21.305067 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0810 03:31:21.305183 (+ 116us) server_negotiation.cc:299] Negotiation successful
0810 03:31:21.305233 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260810 03:31:21.308514 27756 raft_consensus.cc:348] T 00000000000000000000000000000000 P 25668d9bb9f143e3bd9f7e10279bcf73 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25668d9bb9f143e3bd9f7e10279bcf73" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39951 } }
I20260810 03:31:21.308944 27756 raft_consensus.cc:374] T 00000000000000000000000000000000 P 25668d9bb9f143e3bd9f7e10279bcf73 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:31:21.309048 27756 raft_consensus.cc:729] T 00000000000000000000000000000000 P 25668d9bb9f143e3bd9f7e10279bcf73 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25668d9bb9f143e3bd9f7e10279bcf73, State: Initialized, Role: FOLLOWER
I20260810 03:31:21.309685 27756 consensus_queue.cc:260] T 00000000000000000000000000000000 P 25668d9bb9f143e3bd9f7e10279bcf73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25668d9bb9f143e3bd9f7e10279bcf73" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39951 } }
I20260810 03:31:21.309978 27756 raft_consensus.cc:388] T 00000000000000000000000000000000 P 25668d9bb9f143e3bd9f7e10279bcf73 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 03:31:21.310132 27756 raft_consensus.cc:482] T 00000000000000000000000000000000 P 25668d9bb9f143e3bd9f7e10279bcf73 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 03:31:21.310331 27756 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 25668d9bb9f143e3bd9f7e10279bcf73 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:31:21.312826 27756 raft_consensus.cc:504] T 00000000000000000000000000000000 P 25668d9bb9f143e3bd9f7e10279bcf73 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25668d9bb9f143e3bd9f7e10279bcf73" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39951 } }
I20260810 03:31:21.313413 27756 leader_election.cc:302] T 00000000000000000000000000000000 P 25668d9bb9f143e3bd9f7e10279bcf73 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 25668d9bb9f143e3bd9f7e10279bcf73; no voters:
I20260810 03:31:21.314132 27756 leader_election.cc:288] T 00000000000000000000000000000000 P 25668d9bb9f143e3bd9f7e10279bcf73 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 03:31:21.314276 27780 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 25668d9bb9f143e3bd9f7e10279bcf73 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:31:21.315294 27780 raft_consensus.cc:686] T 00000000000000000000000000000000 P 25668d9bb9f143e3bd9f7e10279bcf73 [term 1 LEADER]: Becoming Leader. State: Replica: 25668d9bb9f143e3bd9f7e10279bcf73, State: Running, Role: LEADER
I20260810 03:31:21.316823 27756 sys_catalog.cc:564] T 00000000000000000000000000000000 P 25668d9bb9f143e3bd9f7e10279bcf73 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:31:21.316779 27780 consensus_queue.cc:237] T 00000000000000000000000000000000 P 25668d9bb9f143e3bd9f7e10279bcf73 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25668d9bb9f143e3bd9f7e10279bcf73" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39951 } }
I20260810 03:31:21.319736 27783 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25668d9bb9f143e3bd9f7e10279bcf73 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 25668d9bb9f143e3bd9f7e10279bcf73. Latest consensus state: current_term: 1 leader_uuid: "25668d9bb9f143e3bd9f7e10279bcf73" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25668d9bb9f143e3bd9f7e10279bcf73" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39951 } } }
I20260810 03:31:21.320185 27783 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25668d9bb9f143e3bd9f7e10279bcf73 [sys.catalog]: This master's current role is: LEADER
I20260810 03:31:21.320772 27782 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25668d9bb9f143e3bd9f7e10279bcf73 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "25668d9bb9f143e3bd9f7e10279bcf73" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25668d9bb9f143e3bd9f7e10279bcf73" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39951 } } }
I20260810 03:31:21.321094 27782 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25668d9bb9f143e3bd9f7e10279bcf73 [sys.catalog]: This master's current role is: LEADER
I20260810 03:31:21.321301 27795 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 03:31:21.330294 27795 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 03:31:21.340322 27795 catalog_manager.cc:1347] Generated new cluster ID: 8565eae55e58477a91901e0713d517e3
I20260810 03:31:21.340453 27795 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 03:31:21.357401 27795 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 03:31:21.357985 27795 catalog_manager.cc:1504] Loading token signing keys...
I20260810 03:31:21.370849 27795 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 25668d9bb9f143e3bd9f7e10279bcf73: Generated new TSK 0
I20260810 03:31:21.372087 27795 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 03:31:21.380733 28292 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iBH5/build/asan/bin/kudu pid 27663
W20260810 03:31:21.382030 27747 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:56923
W20260810 03:31:21.382905 28292 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task24iBH5/build/asan/bin/kudu pid 27663: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260810 03:31:21.383071 28292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iBH5/build/asan/bin/kudu with pid 27663
2026-08-10T03:31:21Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (2202 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786332336319544-28292-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 03:31:21 dist-test-slave-vpvm krb5kdc[27813](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 03:31:21 dist-test-slave-vpvm krb5kdc[27813](info): set up 2 sockets
Aug 10 03:31:21 dist-test-slave-vpvm krb5kdc[27813](info): commencing operation
krb5kdc: starting...
W20260810 03:31:22.668130 28292 mini_kdc.cc:121] Time spent starting KDC: real 1.238s user 0.004s sys 0.057s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786332336319544-28292-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786332336319544-28292-0/krb5kdc/test-user.keytab.
Aug 10 03:31:22 dist-test-slave-vpvm krb5kdc[27813](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332682, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T03:31:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T03:31:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786332336319544-28292-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786332336319544-28292-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 03:31:23.059527 28292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iBH5/build/asan/bin/kudu
/tmp/dist-test-task24iBH5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786332336319544-28292-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786332336319544-28292-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786332336319544-28292-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786332336319544-28292-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46127
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786332336319544-28292-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46127
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786332336319544-28292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786332336319544-28292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786332336319544-28292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:31:23.416179 27828 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:31:23.416517 27828 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:31:23.416658 27828 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:31:23.424070 27828 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:31:23.424201 27828 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:31:23.424291 27828 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:31:23.424376 27828 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:31:23.424436 27828 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:31:23.434640 27828 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786332336319544-28292-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786332336319544-28292-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46127
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786332336319544-28292-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46127
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786332336319544-28292-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786332336319544-28292-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:31:23.436024 27828 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:31:23.437381 27828 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:31:23.449345 27840 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:31:23.449415 27837 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:31:23.449369 27842 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:31:23.450416 27828 server_base.cc:1034] running on GCE node
I20260810 03:31:23.451262 27828 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:31:23.452771 27828 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:31:23.454015 27828 hybrid_clock.cc:648] HybridClock initialized: now 1786332683453922 us; error 51 us; skew 500 ppm
I20260810 03:31:23.454385 27828 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:31:23 dist-test-slave-vpvm krb5kdc[27813](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332683, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:31:23.460824 27828 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 03:31:23.462536 27828 webserver.cc:460] Webserver started at http://127.0.0.1:36787/ using document root <none> and password file <none>
I20260810 03:31:23.463344 27828 fs_manager.cc:359] Metadata directory not provided
I20260810 03:31:23.463488 27828 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:31:23.463794 27828 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:31:23.467067 27828 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786332336319544-28292-0/minicluster-data/master-0/data/instance:
uuid: "57c1d0bf93d84104800f8fba3ca92879"
format_stamp: "Formatted at 2026-08-10 03:31:23 on dist-test-slave-vpvm"
server_key: "112c03cbb101a4642951cd6ae801b6e5"
server_key_iv: "fce3e9dc9e0b3ee5cf2f74776aa7a857"
server_key_version: "encryptionkey@0"
I20260810 03:31:23.467890 27828 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786332336319544-28292-0/minicluster-data/master-0/wal/instance:
uuid: "57c1d0bf93d84104800f8fba3ca92879"
format_stamp: "Formatted at 2026-08-10 03:31:23 on dist-test-slave-vpvm"
server_key: "112c03cbb101a4642951cd6ae801b6e5"
server_key_iv: "fce3e9dc9e0b3ee5cf2f74776aa7a857"
server_key_version: "encryptionkey@0"
I20260810 03:31:23.474309 27828 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260810 03:31:23.478083 27852 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:31:23.479419 27828 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 03:31:23.479630 27828 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786332336319544-28292-0/minicluster-data/master-0/data,/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786332336319544-28292-0/minicluster-data/master-0/wal
uuid: "57c1d0bf93d84104800f8fba3ca92879"
format_stamp: "Formatted at 2026-08-10 03:31:23 on dist-test-slave-vpvm"
server_key: "112c03cbb101a4642951cd6ae801b6e5"
server_key_iv: "fce3e9dc9e0b3ee5cf2f74776aa7a857"
server_key_version: "encryptionkey@0"
I20260810 03:31:23.479825 27828 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786332336319544-28292-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786332336319544-28292-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786332336319544-28292-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:31:23.490994 27828 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:31:23.495492 27828 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:31:23.495771 27828 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:31:23.514230 27828 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46127
I20260810 03:31:23.514226 27927 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46127 every 8 connection(s)
I20260810 03:31:23.515724 27828 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786332336319544-28292-0/minicluster-data/master-0/data/info.pb
I20260810 03:31:23.519606 27928 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:31:23.524582 28292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iBH5/build/asan/bin/kudu as pid 27828
I20260810 03:31:23.524747 28292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786332336319544-28292-0/minicluster-data/master-0/wal/instance
I20260810 03:31:23.525172 28292 external_mini_cluster.cc:1428] Setting key 3b0629e19b2b8e4e037be740c22b9ccf
I20260810 03:31:23.532258 27928 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 57c1d0bf93d84104800f8fba3ca92879: Bootstrap starting.
I20260810 03:31:23.536265 27928 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 57c1d0bf93d84104800f8fba3ca92879: Neither blocks nor log segments found. Creating new log.
I20260810 03:31:23.537545 27928 log.cc:824] T 00000000000000000000000000000000 P 57c1d0bf93d84104800f8fba3ca92879: Log is configured to *not* fsync() on all Append() calls
I20260810 03:31:23.540901 27928 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 57c1d0bf93d84104800f8fba3ca92879: No bootstrap required, opened a new log
Aug 10 03:31:23 dist-test-slave-vpvm krb5kdc[27813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332682, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260810 03:31:23.551782 27928 raft_consensus.cc:348] T 00000000000000000000000000000000 P 57c1d0bf93d84104800f8fba3ca92879 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57c1d0bf93d84104800f8fba3ca92879" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46127 } }
I20260810 03:31:23.552273 27928 raft_consensus.cc:374] T 00000000000000000000000000000000 P 57c1d0bf93d84104800f8fba3ca92879 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:31:23.552387 27928 raft_consensus.cc:729] T 00000000000000000000000000000000 P 57c1d0bf93d84104800f8fba3ca92879 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57c1d0bf93d84104800f8fba3ca92879, State: Initialized, Role: FOLLOWER
I20260810 03:31:23.552601 27932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:31:23.527055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41001 (local address 127.0.0.1:46127)
0810 03:31:23.527974 (+ 919us) server_negotiation.cc:206] Beginning negotiation
0810 03:31:23.527990 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 03:31:23.528067 (+ 77us) server_negotiation.cc:407] Connection header received
0810 03:31:23.530605 (+ 2538us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:31:23.530640 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:31:23.531259 (+ 619us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:31:23.531733 (+ 474us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:31:23.533314 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:31:23.535420 (+ 2106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:31:23.536933 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:31:23.538050 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:31:23.538271 (+ 221us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:31:23.542792 (+ 4521us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:31:23.542805 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:31:23.542831 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:31:23.542881 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:31:23.547259 (+ 4378us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:31:23.548163 (+ 904us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:31:23.548179 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:31:23.548190 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:31:23.548446 (+ 256us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:31:23.548922 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:31:23.548927 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:31:23.548930 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:31:23.549813 (+ 883us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:31:23.550171 (+ 358us) server_negotiation.cc:1035] Waiting for connection context
0810 03:31:23.550582 (+ 411us) server_negotiation.cc:299] Negotiation successful
0810 03:31:23.551090 (+ 508us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":644,"thread_start_us":381,"threads_started":1}
I20260810 03:31:23.553081 27928 consensus_queue.cc:260] T 00000000000000000000000000000000 P 57c1d0bf93d84104800f8fba3ca92879 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57c1d0bf93d84104800f8fba3ca92879" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46127 } }
I20260810 03:31:23.553344 27928 raft_consensus.cc:388] T 00000000000000000000000000000000 P 57c1d0bf93d84104800f8fba3ca92879 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 03:31:23.553485 27928 raft_consensus.cc:482] T 00000000000000000000000000000000 P 57c1d0bf93d84104800f8fba3ca92879 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 03:31:23.553663 27928 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 57c1d0bf93d84104800f8fba3ca92879 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:31:23.556892 28292 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 03:31:23.556802 27928 raft_consensus.cc:504] T 00000000000000000000000000000000 P 57c1d0bf93d84104800f8fba3ca92879 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57c1d0bf93d84104800f8fba3ca92879" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46127 } }
I20260810 03:31:23.557382 27928 leader_election.cc:302] T 00000000000000000000000000000000 P 57c1d0bf93d84104800f8fba3ca92879 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 57c1d0bf93d84104800f8fba3ca92879; no voters:
I20260810 03:31:23.558059 27928 leader_election.cc:288] T 00000000000000000000000000000000 P 57c1d0bf93d84104800f8fba3ca92879 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 03:31:23.558189 27935 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 57c1d0bf93d84104800f8fba3ca92879 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:31:23.560309 27928 sys_catalog.cc:564] T 00000000000000000000000000000000 P 57c1d0bf93d84104800f8fba3ca92879 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:31:23.566787 27935 raft_consensus.cc:686] T 00000000000000000000000000000000 P 57c1d0bf93d84104800f8fba3ca92879 [term 1 LEADER]: Becoming Leader. State: Replica: 57c1d0bf93d84104800f8fba3ca92879, State: Running, Role: LEADER
I20260810 03:31:23.567667 27935 consensus_queue.cc:237] T 00000000000000000000000000000000 P 57c1d0bf93d84104800f8fba3ca92879 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57c1d0bf93d84104800f8fba3ca92879" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46127 } }
I20260810 03:31:23.571992 27937 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57c1d0bf93d84104800f8fba3ca92879 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "57c1d0bf93d84104800f8fba3ca92879" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57c1d0bf93d84104800f8fba3ca92879" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46127 } } }
I20260810 03:31:23.572474 27937 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57c1d0bf93d84104800f8fba3ca92879 [sys.catalog]: This master's current role is: LEADER
I20260810 03:31:23.573048 27938 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57c1d0bf93d84104800f8fba3ca92879 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 57c1d0bf93d84104800f8fba3ca92879. Latest consensus state: current_term: 1 leader_uuid: "57c1d0bf93d84104800f8fba3ca92879" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57c1d0bf93d84104800f8fba3ca92879" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46127 } } }
I20260810 03:31:23.573354 27938 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57c1d0bf93d84104800f8fba3ca92879 [sys.catalog]: This master's current role is: LEADER
I20260810 03:31:23.575762 27961 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 03:31:23.582466 27961 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 03:31:23.589666 27932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:31:23.569410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58581 (local address 127.0.0.1:46127)
0810 03:31:23.569684 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0810 03:31:23.569691 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:31:23.570753 (+ 1062us) server_negotiation.cc:407] Connection header received
0810 03:31:23.571077 (+ 324us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:31:23.571083 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:31:23.571171 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:31:23.571258 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:31:23.573700 (+ 2442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:31:23.574915 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:31:23.576656 (+ 1741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:31:23.577510 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:31:23.577654 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:31:23.580915 (+ 3261us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:31:23.580923 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:31:23.580927 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:31:23.580974 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:31:23.585011 (+ 4037us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:31:23.586146 (+ 1135us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:31:23.586153 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:31:23.586160 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:31:23.586357 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:31:23.586880 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:31:23.586886 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:31:23.586890 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:31:23.587424 (+ 534us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:31:23.587691 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0810 03:31:23.588997 (+ 1306us) server_negotiation.cc:299] Negotiation successful
0810 03:31:23.589274 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152}
I20260810 03:31:23.594236 27961 catalog_manager.cc:1347] Generated new cluster ID: f06d72e772a5415c8badc45c68aee9dd
I20260810 03:31:23.594374 27961 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 03:31:23.603533 27961 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 03:31:23.605020 27961 catalog_manager.cc:1504] Loading token signing keys...
I20260810 03:31:23.614450 27961 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 57c1d0bf93d84104800f8fba3ca92879: Generated new TSK 0
I20260810 03:31:23.615391 27961 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 03:31:23.623457 28292 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iBH5/build/asan/bin/kudu pid 27828
I20260810 03:31:23.624707 27921 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:31:23.715070 28292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iBH5/build/asan/bin/kudu with pid 27828
2026-08-10T03:31:23Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (2329 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786332336319544-28292-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 03:31:23 dist-test-slave-vpvm krb5kdc[27984](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 03:31:23 dist-test-slave-vpvm krb5kdc[27984](info): set up 2 sockets
Aug 10 03:31:23 dist-test-slave-vpvm krb5kdc[27984](info): commencing operation
krb5kdc: starting...
W20260810 03:31:24.980544 28292 mini_kdc.cc:121] Time spent starting KDC: real 1.221s user 0.003s sys 0.055s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786332336319544-28292-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786332336319544-28292-0/krb5kdc/test-user.keytab.
Aug 10 03:31:25 dist-test-slave-vpvm krb5kdc[27984](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332685, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T03:31:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T03:31:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786332336319544-28292-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786332336319544-28292-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 03:31:25.335155 28292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iBH5/build/asan/bin/kudu
/tmp/dist-test-task24iBH5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786332336319544-28292-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786332336319544-28292-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786332336319544-28292-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786332336319544-28292-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33055
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786332336319544-28292-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33055
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786332336319544-28292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786332336319544-28292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786332336319544-28292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:31:25.713318 28005 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:31:25.713660 28005 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:31:25.713820 28005 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:31:25.721354 28005 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:31:25.721503 28005 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:31:25.721588 28005 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:31:25.721673 28005 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:31:25.721740 28005 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:31:25.732170 28005 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786332336319544-28292-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786332336319544-28292-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33055
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786332336319544-28292-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33055
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786332336319544-28292-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786332336319544-28292-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:31:25.733577 28005 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:31:25.735028 28005 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:31:25.746907 28011 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:31:25.747016 28013 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:31:25.746999 28015 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:31:25.747799 28005 server_base.cc:1034] running on GCE node
I20260810 03:31:25.748584 28005 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:31:25.750044 28005 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:31:25.751263 28005 hybrid_clock.cc:648] HybridClock initialized: now 1786332685751220 us; error 42 us; skew 500 ppm
I20260810 03:31:25.751631 28005 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:31:25 dist-test-slave-vpvm krb5kdc[27984](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332685, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:31:25.754736 28005 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 03:31:25.756280 28005 webserver.cc:460] Webserver started at http://127.0.0.1:36361/ using document root <none> and password file <none>
I20260810 03:31:25.756973 28005 fs_manager.cc:359] Metadata directory not provided
I20260810 03:31:25.757094 28005 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:31:25.757408 28005 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:31:25.760625 28005 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786332336319544-28292-0/minicluster-data/master-0/data/instance:
uuid: "ffd10c4548c64b63b9c954052f2e3848"
format_stamp: "Formatted at 2026-08-10 03:31:25 on dist-test-slave-vpvm"
server_key: "c05ec97879b17982e37cdb9662b539d1"
server_key_iv: "1dadf29af62988b2b9977a43094ff4c6"
server_key_version: "encryptionkey@0"
I20260810 03:31:25.761415 28005 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786332336319544-28292-0/minicluster-data/master-0/wal/instance:
uuid: "ffd10c4548c64b63b9c954052f2e3848"
format_stamp: "Formatted at 2026-08-10 03:31:25 on dist-test-slave-vpvm"
server_key: "c05ec97879b17982e37cdb9662b539d1"
server_key_iv: "1dadf29af62988b2b9977a43094ff4c6"
server_key_version: "encryptionkey@0"
I20260810 03:31:25.766960 28005 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260810 03:31:25.770538 28025 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:31:25.771979 28005 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260810 03:31:25.772172 28005 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786332336319544-28292-0/minicluster-data/master-0/data,/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786332336319544-28292-0/minicluster-data/master-0/wal
uuid: "ffd10c4548c64b63b9c954052f2e3848"
format_stamp: "Formatted at 2026-08-10 03:31:25 on dist-test-slave-vpvm"
server_key: "c05ec97879b17982e37cdb9662b539d1"
server_key_iv: "1dadf29af62988b2b9977a43094ff4c6"
server_key_version: "encryptionkey@0"
I20260810 03:31:25.772379 28005 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786332336319544-28292-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786332336319544-28292-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786332336319544-28292-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:31:25.785917 28005 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:31:25.790570 28005 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:31:25.790858 28005 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:31:25.809393 28005 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33055
I20260810 03:31:25.809417 28097 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33055 every 8 connection(s)
I20260810 03:31:25.811002 28005 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786332336319544-28292-0/minicluster-data/master-0/data/info.pb
I20260810 03:31:25.815315 28099 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:31:25.820544 28292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iBH5/build/asan/bin/kudu as pid 28005
I20260810 03:31:25.820748 28292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786332336319544-28292-0/minicluster-data/master-0/wal/instance
I20260810 03:31:25.821206 28292 external_mini_cluster.cc:1428] Setting key ea74e352539b53a8c956f1bc489f13fb
I20260810 03:31:25.827278 28099 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ffd10c4548c64b63b9c954052f2e3848: Bootstrap starting.
I20260810 03:31:25.831454 28099 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ffd10c4548c64b63b9c954052f2e3848: Neither blocks nor log segments found. Creating new log.
I20260810 03:31:25.832760 28099 log.cc:824] T 00000000000000000000000000000000 P ffd10c4548c64b63b9c954052f2e3848: Log is configured to *not* fsync() on all Append() calls
I20260810 03:31:25.836383 28105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:31:25.823114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48301 (local address 127.0.0.1:33055)
0810 03:31:25.823983 (+ 869us) server_negotiation.cc:206] Beginning negotiation
0810 03:31:25.823996 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 03:31:25.824043 (+ 47us) server_negotiation.cc:407] Connection header received
0810 03:31:25.825878 (+ 1835us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:31:25.825906 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:31:25.826405 (+ 499us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:31:25.826764 (+ 359us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:31:25.827469 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:31:25.829213 (+ 1744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:31:25.831289 (+ 2076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:31:25.832161 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:31:25.832320 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:31:25.833661 (+ 1341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:31:25.833672 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:31:25.833707 (+ 35us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:31:25.834529 (+ 822us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:31:25.834615 (+ 86us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:31:25.834644 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:31:25.834757 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0810 03:31:25.835019 (+ 262us) server_negotiation.cc:299] Negotiation successful
0810 03:31:25.835306 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":630,"thread_start_us":365,"threads_started":1}
I20260810 03:31:25.836460 28099 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ffd10c4548c64b63b9c954052f2e3848: No bootstrap required, opened a new log
I20260810 03:31:25.840852 28292 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 03:31:25.846640 28099 raft_consensus.cc:348] T 00000000000000000000000000000000 P ffd10c4548c64b63b9c954052f2e3848 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffd10c4548c64b63b9c954052f2e3848" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33055 } }
I20260810 03:31:25.847157 28099 raft_consensus.cc:374] T 00000000000000000000000000000000 P ffd10c4548c64b63b9c954052f2e3848 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:31:25.847270 28099 raft_consensus.cc:729] T 00000000000000000000000000000000 P ffd10c4548c64b63b9c954052f2e3848 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ffd10c4548c64b63b9c954052f2e3848, State: Initialized, Role: FOLLOWER
I20260810 03:31:25.848032 28099 consensus_queue.cc:260] T 00000000000000000000000000000000 P ffd10c4548c64b63b9c954052f2e3848 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffd10c4548c64b63b9c954052f2e3848" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33055 } }
I20260810 03:31:25.848328 28099 raft_consensus.cc:388] T 00000000000000000000000000000000 P ffd10c4548c64b63b9c954052f2e3848 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 03:31:25.848482 28099 raft_consensus.cc:482] T 00000000000000000000000000000000 P ffd10c4548c64b63b9c954052f2e3848 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 03:31:25.848695 28099 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ffd10c4548c64b63b9c954052f2e3848 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:31:25.851704 28099 raft_consensus.cc:504] T 00000000000000000000000000000000 P ffd10c4548c64b63b9c954052f2e3848 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffd10c4548c64b63b9c954052f2e3848" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33055 } }
I20260810 03:31:25.852299 28099 leader_election.cc:302] T 00000000000000000000000000000000 P ffd10c4548c64b63b9c954052f2e3848 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ffd10c4548c64b63b9c954052f2e3848; no voters:
I20260810 03:31:25.852979 28099 leader_election.cc:288] T 00000000000000000000000000000000 P ffd10c4548c64b63b9c954052f2e3848 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 03:31:25.853101 28119 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ffd10c4548c64b63b9c954052f2e3848 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:31:25.853423 28105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:31:25.846731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49539 (local address 127.0.0.1:33055)
0810 03:31:25.846941 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0810 03:31:25.846948 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:31:25.847344 (+ 396us) server_negotiation.cc:407] Connection header received
0810 03:31:25.847564 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:31:25.847570 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:31:25.847660 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:31:25.847782 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:31:25.848252 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:31:25.849323 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:31:25.850766 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:31:25.851448 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:31:25.851593 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:31:25.852609 (+ 1016us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:31:25.852615 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:31:25.852618 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:31:25.852639 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:31:25.852683 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:31:25.852708 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:31:25.852798 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0810 03:31:25.852948 (+ 150us) server_negotiation.cc:299] Negotiation successful
0810 03:31:25.853054 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260810 03:31:25.855659 28099 sys_catalog.cc:564] T 00000000000000000000000000000000 P ffd10c4548c64b63b9c954052f2e3848 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:31:25.860083 28119 raft_consensus.cc:686] T 00000000000000000000000000000000 P ffd10c4548c64b63b9c954052f2e3848 [term 1 LEADER]: Becoming Leader. State: Replica: ffd10c4548c64b63b9c954052f2e3848, State: Running, Role: LEADER
I20260810 03:31:25.860895 28119 consensus_queue.cc:237] T 00000000000000000000000000000000 P ffd10c4548c64b63b9c954052f2e3848 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffd10c4548c64b63b9c954052f2e3848" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33055 } }
I20260810 03:31:25.864634 28122 sys_catalog.cc:455] T 00000000000000000000000000000000 P ffd10c4548c64b63b9c954052f2e3848 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ffd10c4548c64b63b9c954052f2e3848. Latest consensus state: current_term: 1 leader_uuid: "ffd10c4548c64b63b9c954052f2e3848" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffd10c4548c64b63b9c954052f2e3848" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33055 } } }
I20260810 03:31:25.864545 28120 sys_catalog.cc:455] T 00000000000000000000000000000000 P ffd10c4548c64b63b9c954052f2e3848 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ffd10c4548c64b63b9c954052f2e3848" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffd10c4548c64b63b9c954052f2e3848" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33055 } } }
I20260810 03:31:25.865123 28120 sys_catalog.cc:458] T 00000000000000000000000000000000 P ffd10c4548c64b63b9c954052f2e3848 [sys.catalog]: This master's current role is: LEADER
I20260810 03:31:25.865810 28122 sys_catalog.cc:458] T 00000000000000000000000000000000 P ffd10c4548c64b63b9c954052f2e3848 [sys.catalog]: This master's current role is: LEADER
I20260810 03:31:25.866102 28137 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 03:31:25.871723 28137 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 03:31:25.881258 28137 catalog_manager.cc:1347] Generated new cluster ID: f58bdd05db734d28916d569a31cb1854
I20260810 03:31:25.881397 28137 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 03:31:25.898105 28137 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 03:31:25.899618 28137 catalog_manager.cc:1504] Loading token signing keys...
I20260810 03:31:25.916116 28137 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ffd10c4548c64b63b9c954052f2e3848: Generated new TSK 0
I20260810 03:31:25.917353 28137 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 03:31:25.930101 28292 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iBH5/build/asan/bin/kudu pid 28005
W20260810 03:31:25.931272 28092 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:48301
W20260810 03:31:25.932238 28292 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task24iBH5/build/asan/bin/kudu pid 28005: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260810 03:31:25.932390 28292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iBH5/build/asan/bin/kudu with pid 28005
2026-08-10T03:31:25Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (2217 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786332336319544-28292-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 03:31:26 dist-test-slave-vpvm krb5kdc[28153](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 03:31:26 dist-test-slave-vpvm krb5kdc[28153](info): set up 2 sockets
Aug 10 03:31:26 dist-test-slave-vpvm krb5kdc[28153](info): commencing operation
krb5kdc: starting...
W20260810 03:31:27.193903 28292 mini_kdc.cc:121] Time spent starting KDC: real 1.217s user 0.000s sys 0.057s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786332336319544-28292-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786332336319544-28292-0/krb5kdc/test-user.keytab.
Aug 10 03:31:27 dist-test-slave-vpvm krb5kdc[28153](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332687, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T03:31:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T03:31:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786332336319544-28292-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786332336319544-28292-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 03:31:27.551504 28292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iBH5/build/asan/bin/kudu
/tmp/dist-test-task24iBH5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786332336319544-28292-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786332336319544-28292-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786332336319544-28292-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786332336319544-28292-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39321
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786332336319544-28292-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39321
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786332336319544-28292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786332336319544-28292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786332336319544-28292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:31:27.909410 28172 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:31:27.909793 28172 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:31:27.909981 28172 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:31:27.917595 28172 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:31:27.917755 28172 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:31:27.917852 28172 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:31:27.917984 28172 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:31:27.918054 28172 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:31:27.929172 28172 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786332336319544-28292-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786332336319544-28292-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39321
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786332336319544-28292-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39321
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786332336319544-28292-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786332336319544-28292-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:31:27.930732 28172 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:31:27.932132 28172 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:31:27.944518 28179 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:31:27.944558 28180 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:31:27.944602 28183 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:31:27.945504 28172 server_base.cc:1034] running on GCE node
I20260810 03:31:27.946420 28172 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:31:27.947958 28172 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:31:27.949213 28172 hybrid_clock.cc:648] HybridClock initialized: now 1786332687949171 us; error 30 us; skew 500 ppm
I20260810 03:31:27.949569 28172 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:31:27 dist-test-slave-vpvm krb5kdc[28153](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332687, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:31:27.952679 28172 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 03:31:27.954242 28172 webserver.cc:460] Webserver started at http://127.0.0.1:34621/ using document root <none> and password file <none>
I20260810 03:31:27.954926 28172 fs_manager.cc:359] Metadata directory not provided
I20260810 03:31:27.955044 28172 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:31:27.955358 28172 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:31:27.958508 28172 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786332336319544-28292-0/minicluster-data/master-0/data/instance:
uuid: "81c6a625569448ba82154fd202e6845f"
format_stamp: "Formatted at 2026-08-10 03:31:27 on dist-test-slave-vpvm"
server_key: "442cfa26ca60664ad01cce11d02d8c69"
server_key_iv: "11d461987ee95d7e754c429e6683a6e5"
server_key_version: "encryptionkey@0"
I20260810 03:31:27.959321 28172 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786332336319544-28292-0/minicluster-data/master-0/wal/instance:
uuid: "81c6a625569448ba82154fd202e6845f"
format_stamp: "Formatted at 2026-08-10 03:31:27 on dist-test-slave-vpvm"
server_key: "442cfa26ca60664ad01cce11d02d8c69"
server_key_iv: "11d461987ee95d7e754c429e6683a6e5"
server_key_version: "encryptionkey@0"
I20260810 03:31:27.964653 28172 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260810 03:31:27.968189 28192 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:31:27.969834 28172 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 03:31:27.970078 28172 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786332336319544-28292-0/minicluster-data/master-0/data,/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786332336319544-28292-0/minicluster-data/master-0/wal
uuid: "81c6a625569448ba82154fd202e6845f"
format_stamp: "Formatted at 2026-08-10 03:31:27 on dist-test-slave-vpvm"
server_key: "442cfa26ca60664ad01cce11d02d8c69"
server_key_iv: "11d461987ee95d7e754c429e6683a6e5"
server_key_version: "encryptionkey@0"
I20260810 03:31:27.970276 28172 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786332336319544-28292-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786332336319544-28292-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786332336319544-28292-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:31:27.975973 28172 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:31:27.980190 28172 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:31:27.980474 28172 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:31:27.998476 28172 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39321
I20260810 03:31:27.998529 28264 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39321 every 8 connection(s)
I20260810 03:31:27.999862 28172 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786332336319544-28292-0/minicluster-data/master-0/data/info.pb
I20260810 03:31:28.003679 28265 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:31:28.007530 28292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iBH5/build/asan/bin/kudu as pid 28172
I20260810 03:31:28.007719 28292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786332336319544-28292-0/minicluster-data/master-0/wal/instance
I20260810 03:31:28.008270 28292 external_mini_cluster.cc:1428] Setting key 6e06d00ce04a4c60fa36e43bfa07a643
I20260810 03:31:28.015923 28268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:31:28.010204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57353 (local address 127.0.0.1:39321)
0810 03:31:28.011007 (+ 803us) server_negotiation.cc:206] Beginning negotiation
0810 03:31:28.011019 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 03:31:28.011061 (+ 42us) server_negotiation.cc:407] Connection header received
0810 03:31:28.012685 (+ 1624us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:31:28.012707 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:31:28.013055 (+ 348us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:31:28.013390 (+ 335us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:31:28.014441 (+ 1051us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:31:28.014447 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:31:28.014460 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:31:28.014474 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:31:28.014537 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:31:28.014556 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:31:28.014726 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0810 03:31:28.014834 (+ 108us) server_negotiation.cc:299] Negotiation successful
0810 03:31:28.015012 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":603,"thread_start_us":364,"threads_started":1}
I20260810 03:31:28.016681 28265 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 81c6a625569448ba82154fd202e6845f: Bootstrap starting.
I20260810 03:31:28.019608 28292 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 03:31:28.020120 28265 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 81c6a625569448ba82154fd202e6845f: Neither blocks nor log segments found. Creating new log.
I20260810 03:31:28.021123 28265 log.cc:824] T 00000000000000000000000000000000 P 81c6a625569448ba82154fd202e6845f: Log is configured to *not* fsync() on all Append() calls
I20260810 03:31:28.024648 28265 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 81c6a625569448ba82154fd202e6845f: No bootstrap required, opened a new log
I20260810 03:31:28.026585 28268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:31:28.024215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40555 (local address 127.0.0.1:39321)
0810 03:31:28.024387 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0810 03:31:28.024392 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:31:28.024703 (+ 311us) server_negotiation.cc:407] Connection header received
0810 03:31:28.024857 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:31:28.024862 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:31:28.024922 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:31:28.025045 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:31:28.025896 (+ 851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:31:28.025903 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:31:28.025911 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:31:28.025930 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:31:28.026024 (+ 94us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:31:28.026048 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:31:28.026166 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0810 03:31:28.026243 (+ 77us) server_negotiation.cc:299] Negotiation successful
0810 03:31:28.026302 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260810 03:31:28.033064 28265 raft_consensus.cc:348] T 00000000000000000000000000000000 P 81c6a625569448ba82154fd202e6845f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81c6a625569448ba82154fd202e6845f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39321 } }
I20260810 03:31:28.033453 28265 raft_consensus.cc:374] T 00000000000000000000000000000000 P 81c6a625569448ba82154fd202e6845f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:31:28.033535 28265 raft_consensus.cc:729] T 00000000000000000000000000000000 P 81c6a625569448ba82154fd202e6845f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81c6a625569448ba82154fd202e6845f, State: Initialized, Role: FOLLOWER
I20260810 03:31:28.034087 28265 consensus_queue.cc:260] T 00000000000000000000000000000000 P 81c6a625569448ba82154fd202e6845f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81c6a625569448ba82154fd202e6845f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39321 } }
I20260810 03:31:28.034307 28265 raft_consensus.cc:388] T 00000000000000000000000000000000 P 81c6a625569448ba82154fd202e6845f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 03:31:28.034409 28265 raft_consensus.cc:482] T 00000000000000000000000000000000 P 81c6a625569448ba82154fd202e6845f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 03:31:28.034541 28265 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 81c6a625569448ba82154fd202e6845f [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:31:28.036679 28265 raft_consensus.cc:504] T 00000000000000000000000000000000 P 81c6a625569448ba82154fd202e6845f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81c6a625569448ba82154fd202e6845f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39321 } }
I20260810 03:31:28.037168 28265 leader_election.cc:302] T 00000000000000000000000000000000 P 81c6a625569448ba82154fd202e6845f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 81c6a625569448ba82154fd202e6845f; no voters:
I20260810 03:31:28.037827 28265 leader_election.cc:288] T 00000000000000000000000000000000 P 81c6a625569448ba82154fd202e6845f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 03:31:28.037972 28279 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 81c6a625569448ba82154fd202e6845f [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:31:28.038579 28279 raft_consensus.cc:686] T 00000000000000000000000000000000 P 81c6a625569448ba82154fd202e6845f [term 1 LEADER]: Becoming Leader. State: Replica: 81c6a625569448ba82154fd202e6845f, State: Running, Role: LEADER
I20260810 03:31:28.039103 28279 consensus_queue.cc:237] T 00000000000000000000000000000000 P 81c6a625569448ba82154fd202e6845f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81c6a625569448ba82154fd202e6845f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39321 } }
I20260810 03:31:28.039897 28265 sys_catalog.cc:564] T 00000000000000000000000000000000 P 81c6a625569448ba82154fd202e6845f [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:31:28.042181 28280 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81c6a625569448ba82154fd202e6845f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "81c6a625569448ba82154fd202e6845f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81c6a625569448ba82154fd202e6845f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39321 } } }
I20260810 03:31:28.042207 28282 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81c6a625569448ba82154fd202e6845f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 81c6a625569448ba82154fd202e6845f. Latest consensus state: current_term: 1 leader_uuid: "81c6a625569448ba82154fd202e6845f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81c6a625569448ba82154fd202e6845f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39321 } } }
I20260810 03:31:28.043135 28282 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81c6a625569448ba82154fd202e6845f [sys.catalog]: This master's current role is: LEADER
I20260810 03:31:28.042788 28280 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81c6a625569448ba82154fd202e6845f [sys.catalog]: This master's current role is: LEADER
I20260810 03:31:28.044049 28298 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 03:31:28.049612 28298 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 03:31:28.059053 28298 catalog_manager.cc:1347] Generated new cluster ID: dcbed877d48641dc80799aae0c5aa98d
I20260810 03:31:28.059186 28298 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 03:31:28.076529 28298 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 03:31:28.076972 28298 catalog_manager.cc:1504] Loading token signing keys...
I20260810 03:31:28.085259 28298 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 81c6a625569448ba82154fd202e6845f: Generated new TSK 0
I20260810 03:31:28.086120 28298 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 03:31:28.097330 28292 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iBH5/build/asan/bin/kudu pid 28172
W20260810 03:31:28.098419 28258 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:57353
W20260810 03:31:28.099077 28292 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task24iBH5/build/asan/bin/kudu pid 28172: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260810 03:31:28.099210 28292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iBH5/build/asan/bin/kudu with pid 28172
2026-08-10T03:31:28Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (2163 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786332336319544-28292-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 03:31:28 dist-test-slave-vpvm krb5kdc[28315](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 03:31:28 dist-test-slave-vpvm krb5kdc[28315](info): set up 2 sockets
Aug 10 03:31:28 dist-test-slave-vpvm krb5kdc[28315](info): commencing operation
krb5kdc: starting...
W20260810 03:31:29.362790 28292 mini_kdc.cc:121] Time spent starting KDC: real 1.222s user 0.001s sys 0.054s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786332336319544-28292-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786332336319544-28292-0/krb5kdc/test-user.keytab.
Aug 10 03:31:29 dist-test-slave-vpvm krb5kdc[28315](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332689, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T03:31:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T03:31:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786332336319544-28292-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786332336319544-28292-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 03:31:29.717095 28292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iBH5/build/asan/bin/kudu
/tmp/dist-test-task24iBH5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786332336319544-28292-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786332336319544-28292-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786332336319544-28292-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786332336319544-28292-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46239
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786332336319544-28292-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39947
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46239
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786332336319544-28292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786332336319544-28292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786332336319544-28292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:31:30.085659 28333 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:31:30.086030 28333 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:31:30.086193 28333 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:31:30.093791 28333 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:31:30.093931 28333 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:31:30.094033 28333 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:31:30.094132 28333 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:31:30.094192 28333 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:31:30.104913 28333 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39947
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786332336319544-28292-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786332336319544-28292-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46239
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786332336319544-28292-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46239
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786332336319544-28292-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786332336319544-28292-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:31:30.106360 28333 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:31:30.107749 28333 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:31:30.119833 28340 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:31:30.119902 28341 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:31:30.119859 28344 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:31:30.121351 28333 server_base.cc:1034] running on GCE node
I20260810 03:31:30.122236 28333 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:31:30.123838 28333 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:31:30.125092 28333 hybrid_clock.cc:648] HybridClock initialized: now 1786332690125029 us; error 49 us; skew 500 ppm
I20260810 03:31:30.125479 28333 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:31:30 dist-test-slave-vpvm krb5kdc[28315](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332690, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:31:30.132180 28333 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 03:31:30.133879 28333 webserver.cc:460] Webserver started at http://127.0.0.1:36439/ using document root <none> and password file <none>
I20260810 03:31:30.134663 28333 fs_manager.cc:359] Metadata directory not provided
I20260810 03:31:30.134773 28333 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:31:30.135043 28333 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:31:30.138269 28333 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786332336319544-28292-0/minicluster-data/master-0/data/instance:
uuid: "83346ae54b884288acefb522e43bbd3a"
format_stamp: "Formatted at 2026-08-10 03:31:30 on dist-test-slave-vpvm"
server_key: "4372043873eed930b7e513ce9785d740"
server_key_iv: "01d26c06d6478579137db1c4e5ed2174"
server_key_version: "encryptionkey@0"
I20260810 03:31:30.139063 28333 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786332336319544-28292-0/minicluster-data/master-0/wal/instance:
uuid: "83346ae54b884288acefb522e43bbd3a"
format_stamp: "Formatted at 2026-08-10 03:31:30 on dist-test-slave-vpvm"
server_key: "4372043873eed930b7e513ce9785d740"
server_key_iv: "01d26c06d6478579137db1c4e5ed2174"
server_key_version: "encryptionkey@0"
I20260810 03:31:30.144675 28333 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260810 03:31:30.148398 28353 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:31:30.149739 28333 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260810 03:31:30.149904 28333 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786332336319544-28292-0/minicluster-data/master-0/data,/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786332336319544-28292-0/minicluster-data/master-0/wal
uuid: "83346ae54b884288acefb522e43bbd3a"
format_stamp: "Formatted at 2026-08-10 03:31:30 on dist-test-slave-vpvm"
server_key: "4372043873eed930b7e513ce9785d740"
server_key_iv: "01d26c06d6478579137db1c4e5ed2174"
server_key_version: "encryptionkey@0"
I20260810 03:31:30.150139 28333 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786332336319544-28292-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786332336319544-28292-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786332336319544-28292-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:31:30.163774 28333 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:31:30.168022 28333 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:31:30.168299 28333 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:31:30.186113 28333 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46239
I20260810 03:31:30.186183 28432 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46239 every 8 connection(s)
I20260810 03:31:30.187484 28333 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786332336319544-28292-0/minicluster-data/master-0/data/info.pb
I20260810 03:31:30.191375 28433 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:31:30.193941 28292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iBH5/build/asan/bin/kudu as pid 28333
I20260810 03:31:30.194123 28292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786332336319544-28292-0/minicluster-data/master-0/wal/instance
I20260810 03:31:30.194538 28292 external_mini_cluster.cc:1428] Setting key 69582e1259c4f31a9dcf39e4bdaffd6a
I20260810 03:31:30.203173 28433 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 83346ae54b884288acefb522e43bbd3a: Bootstrap starting.
I20260810 03:31:30.206521 28433 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 83346ae54b884288acefb522e43bbd3a: Neither blocks nor log segments found. Creating new log.
I20260810 03:31:30.207814 28433 log.cc:824] T 00000000000000000000000000000000 P 83346ae54b884288acefb522e43bbd3a: Log is configured to *not* fsync() on all Append() calls
Aug 10 03:31:30 dist-test-slave-vpvm krb5kdc[28315](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332689, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260810 03:31:30.211925 28433 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 83346ae54b884288acefb522e43bbd3a: No bootstrap required, opened a new log
I20260810 03:31:30.218732 28437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:31:30.196398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.27:43451 (local address 127.0.0.1:46239)
0810 03:31:30.197268 (+ 870us) server_negotiation.cc:206] Beginning negotiation
0810 03:31:30.197282 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 03:31:30.197327 (+ 45us) server_negotiation.cc:407] Connection header received
0810 03:31:30.199077 (+ 1750us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:31:30.199099 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:31:30.199485 (+ 386us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:31:30.199813 (+ 328us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:31:30.201272 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:31:30.202623 (+ 1351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:31:30.203938 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:31:30.204592 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:31:30.204764 (+ 172us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:31:30.209770 (+ 5006us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:31:30.209781 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:31:30.209802 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:31:30.209853 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:31:30.213771 (+ 3918us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:31:30.214718 (+ 947us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:31:30.214727 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:31:30.214737 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:31:30.214964 (+ 227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:31:30.215379 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:31:30.215384 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:31:30.215387 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:31:30.216128 (+ 741us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:31:30.216467 (+ 339us) server_negotiation.cc:1035] Waiting for connection context
0810 03:31:30.216847 (+ 380us) server_negotiation.cc:299] Negotiation successful
0810 03:31:30.217238 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":664,"thread_start_us":346,"threads_started":1}
I20260810 03:31:30.221789 28433 raft_consensus.cc:348] T 00000000000000000000000000000000 P 83346ae54b884288acefb522e43bbd3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83346ae54b884288acefb522e43bbd3a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46239 } }
I20260810 03:31:30.222234 28433 raft_consensus.cc:374] T 00000000000000000000000000000000 P 83346ae54b884288acefb522e43bbd3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:31:30.222337 28433 raft_consensus.cc:729] T 00000000000000000000000000000000 P 83346ae54b884288acefb522e43bbd3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83346ae54b884288acefb522e43bbd3a, State: Initialized, Role: FOLLOWER
I20260810 03:31:30.222875 28292 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 03:31:30.222988 28433 consensus_queue.cc:260] T 00000000000000000000000000000000 P 83346ae54b884288acefb522e43bbd3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83346ae54b884288acefb522e43bbd3a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46239 } }
I20260810 03:31:30.223272 28433 raft_consensus.cc:388] T 00000000000000000000000000000000 P 83346ae54b884288acefb522e43bbd3a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 03:31:30.223404 28433 raft_consensus.cc:482] T 00000000000000000000000000000000 P 83346ae54b884288acefb522e43bbd3a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 03:31:30.223562 28433 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 83346ae54b884288acefb522e43bbd3a [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:31:30.226795 28433 raft_consensus.cc:504] T 00000000000000000000000000000000 P 83346ae54b884288acefb522e43bbd3a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83346ae54b884288acefb522e43bbd3a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46239 } }
I20260810 03:31:30.227415 28433 leader_election.cc:302] T 00000000000000000000000000000000 P 83346ae54b884288acefb522e43bbd3a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 83346ae54b884288acefb522e43bbd3a; no voters:
I20260810 03:31:30.228106 28433 leader_election.cc:288] T 00000000000000000000000000000000 P 83346ae54b884288acefb522e43bbd3a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 03:31:30.228274 28445 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 83346ae54b884288acefb522e43bbd3a [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:31:30.229091 28445 raft_consensus.cc:686] T 00000000000000000000000000000000 P 83346ae54b884288acefb522e43bbd3a [term 1 LEADER]: Becoming Leader. State: Replica: 83346ae54b884288acefb522e43bbd3a, State: Running, Role: LEADER
I20260810 03:31:30.229823 28445 consensus_queue.cc:237] T 00000000000000000000000000000000 P 83346ae54b884288acefb522e43bbd3a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83346ae54b884288acefb522e43bbd3a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46239 } }
I20260810 03:31:30.231829 28433 sys_catalog.cc:564] T 00000000000000000000000000000000 P 83346ae54b884288acefb522e43bbd3a [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:31:30.233282 28451 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83346ae54b884288acefb522e43bbd3a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "83346ae54b884288acefb522e43bbd3a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83346ae54b884288acefb522e43bbd3a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46239 } } }
I20260810 03:31:30.233688 28451 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83346ae54b884288acefb522e43bbd3a [sys.catalog]: This master's current role is: LEADER
I20260810 03:31:30.234072 28449 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83346ae54b884288acefb522e43bbd3a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 83346ae54b884288acefb522e43bbd3a. Latest consensus state: current_term: 1 leader_uuid: "83346ae54b884288acefb522e43bbd3a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83346ae54b884288acefb522e43bbd3a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46239 } } }
I20260810 03:31:30.234355 28449 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83346ae54b884288acefb522e43bbd3a [sys.catalog]: This master's current role is: LEADER
I20260810 03:31:30.236198 28455 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 03:31:30.241465 28455 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 03:31:30.255810 28437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:31:30.228233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.27:41105 (local address 127.0.0.1:46239)
0810 03:31:30.228441 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0810 03:31:30.228448 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:31:30.230573 (+ 2125us) server_negotiation.cc:407] Connection header received
0810 03:31:30.230781 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:31:30.230787 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:31:30.230965 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:31:30.231082 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:31:30.235612 (+ 4530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:31:30.236810 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:31:30.238775 (+ 1965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:31:30.239543 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:31:30.239664 (+ 121us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:31:30.243539 (+ 3875us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:31:30.243548 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:31:30.243554 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:31:30.243608 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:31:30.249232 (+ 5624us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:31:30.250611 (+ 1379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:31:30.250618 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:31:30.250623 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:31:30.250804 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:31:30.251405 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:31:30.251411 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:31:30.251415 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:31:30.251888 (+ 473us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:31:30.252168 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0810 03:31:30.255126 (+ 2958us) server_negotiation.cc:299] Negotiation successful
0810 03:31:30.255366 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260810 03:31:30.259017 28455 catalog_manager.cc:1347] Generated new cluster ID: b86e23e0c8ad4b86898ab312bced2729
I20260810 03:31:30.259150 28455 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 03:31:30.275030 28455 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 03:31:30.276453 28455 catalog_manager.cc:1504] Loading token signing keys...
I20260810 03:31:30.283629 28455 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 83346ae54b884288acefb522e43bbd3a: Generated new TSK 0
I20260810 03:31:30.284588 28455 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 03:31:30.289875 28292 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iBH5/build/asan/bin/kudu pid 28333
I20260810 03:31:30.291041 28423 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:31:30.381090 28292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iBH5/build/asan/bin/kudu with pid 28333
W20260810 03:31:30.405361 28312 connection.cc:535] client connection to 127.0.0.1:46239 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46239): Connection reset by peer (error 104)
2026-08-10T03:31:30Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (2284 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786332336319544-28292-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 03:31:30 dist-test-slave-vpvm krb5kdc[28479](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 03:31:30 dist-test-slave-vpvm krb5kdc[28479](info): set up 2 sockets
Aug 10 03:31:30 dist-test-slave-vpvm krb5kdc[28479](info): commencing operation
krb5kdc: starting...
W20260810 03:31:31.643196 28292 mini_kdc.cc:121] Time spent starting KDC: real 1.219s user 0.004s sys 0.054s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786332336319544-28292-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786332336319544-28292-0/krb5kdc/test-user.keytab.
Aug 10 03:31:31 dist-test-slave-vpvm krb5kdc[28479](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332691, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T03:31:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T03:31:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786332336319544-28292-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786332336319544-28292-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 03:31:31.996806 28292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iBH5/build/asan/bin/kudu
/tmp/dist-test-task24iBH5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786332336319544-28292-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786332336319544-28292-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786332336319544-28292-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786332336319544-28292-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39505
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786332336319544-28292-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41985
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39505
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786332336319544-28292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786332336319544-28292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786332336319544-28292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:31:32.355388 28496 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:31:32.355737 28496 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:31:32.355880 28496 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:31:32.363512 28496 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:31:32.363667 28496 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:31:32.363750 28496 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:31:32.363843 28496 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:31:32.363905 28496 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:31:32.374444 28496 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41985
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786332336319544-28292-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786332336319544-28292-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39505
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786332336319544-28292-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39505
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786332336319544-28292-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786332336319544-28292-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:31:32.375878 28496 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:31:32.377267 28496 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:31:32.389662 28503 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:31:32.389698 28514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:31:32.389737 28504 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:31:32.390828 28496 server_base.cc:1034] running on GCE node
I20260810 03:31:32.391680 28496 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:31:32.393214 28496 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:31:32.394431 28496 hybrid_clock.cc:648] HybridClock initialized: now 1786332692394372 us; error 58 us; skew 500 ppm
I20260810 03:31:32.394796 28496 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:31:32 dist-test-slave-vpvm krb5kdc[28479](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332692, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:31:32.398028 28496 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 03:31:32.399749 28496 webserver.cc:460] Webserver started at http://127.0.0.1:42091/ using document root <none> and password file <none>
I20260810 03:31:32.400511 28496 fs_manager.cc:359] Metadata directory not provided
I20260810 03:31:32.400645 28496 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:31:32.401000 28496 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:31:32.404752 28496 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786332336319544-28292-0/minicluster-data/master-0/data/instance:
uuid: "6c2afa256e814479bd20f250d14ec80b"
format_stamp: "Formatted at 2026-08-10 03:31:32 on dist-test-slave-vpvm"
server_key: "a5ddaf3429f1a395ea70108971e83c28"
server_key_iv: "5f7b650094b7d752f037c40427b2e8a7"
server_key_version: "encryptionkey@0"
I20260810 03:31:32.405654 28496 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786332336319544-28292-0/minicluster-data/master-0/wal/instance:
uuid: "6c2afa256e814479bd20f250d14ec80b"
format_stamp: "Formatted at 2026-08-10 03:31:32 on dist-test-slave-vpvm"
server_key: "a5ddaf3429f1a395ea70108971e83c28"
server_key_iv: "5f7b650094b7d752f037c40427b2e8a7"
server_key_version: "encryptionkey@0"
I20260810 03:31:32.411695 28496 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.000s sys 0.004s
I20260810 03:31:32.415949 28524 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:31:32.417636 28496 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260810 03:31:32.417837 28496 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786332336319544-28292-0/minicluster-data/master-0/data,/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786332336319544-28292-0/minicluster-data/master-0/wal
uuid: "6c2afa256e814479bd20f250d14ec80b"
format_stamp: "Formatted at 2026-08-10 03:31:32 on dist-test-slave-vpvm"
server_key: "a5ddaf3429f1a395ea70108971e83c28"
server_key_iv: "5f7b650094b7d752f037c40427b2e8a7"
server_key_version: "encryptionkey@0"
I20260810 03:31:32.418131 28496 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786332336319544-28292-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786332336319544-28292-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786332336319544-28292-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:31:32.438983 28496 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:31:32.445273 28496 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:31:32.445590 28496 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:31:32.464183 28496 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39505
I20260810 03:31:32.464221 28601 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39505 every 8 connection(s)
I20260810 03:31:32.465572 28496 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786332336319544-28292-0/minicluster-data/master-0/data/info.pb
I20260810 03:31:32.469465 28602 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:31:32.473577 28292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iBH5/build/asan/bin/kudu as pid 28496
I20260810 03:31:32.473742 28292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786332336319544-28292-0/minicluster-data/master-0/wal/instance
I20260810 03:31:32.474222 28292 external_mini_cluster.cc:1428] Setting key 8ff7851e03db89bfc05a3aa35bc21602
I20260810 03:31:32.480962 28602 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c2afa256e814479bd20f250d14ec80b: Bootstrap starting.
I20260810 03:31:32.484802 28602 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6c2afa256e814479bd20f250d14ec80b: Neither blocks nor log segments found. Creating new log.
I20260810 03:31:32.485904 28602 log.cc:824] T 00000000000000000000000000000000 P 6c2afa256e814479bd20f250d14ec80b: Log is configured to *not* fsync() on all Append() calls
I20260810 03:31:32.489166 28605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:31:32.476165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.27:35743 (local address 127.0.0.1:39505)
0810 03:31:32.476932 (+ 767us) server_negotiation.cc:206] Beginning negotiation
0810 03:31:32.476944 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 03:31:32.476990 (+ 46us) server_negotiation.cc:407] Connection header received
0810 03:31:32.479379 (+ 2389us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:31:32.479406 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:31:32.479853 (+ 447us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:31:32.480175 (+ 322us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:31:32.480817 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:31:32.482262 (+ 1445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:31:32.484152 (+ 1890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:31:32.484860 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:31:32.485043 (+ 183us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:31:32.486312 (+ 1269us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:31:32.486325 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:31:32.486347 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:31:32.487072 (+ 725us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:31:32.487160 (+ 88us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:31:32.487197 (+ 37us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:31:32.487335 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0810 03:31:32.487662 (+ 327us) server_negotiation.cc:299] Negotiation successful
0810 03:31:32.487848 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":548,"thread_start_us":306,"threads_started":1}
I20260810 03:31:32.489550 28602 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c2afa256e814479bd20f250d14ec80b: No bootstrap required, opened a new log
I20260810 03:31:32.493206 28292 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 03:31:32.497859 28602 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6c2afa256e814479bd20f250d14ec80b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c2afa256e814479bd20f250d14ec80b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39505 } }
I20260810 03:31:32.498402 28602 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6c2afa256e814479bd20f250d14ec80b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:31:32.498533 28602 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6c2afa256e814479bd20f250d14ec80b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c2afa256e814479bd20f250d14ec80b, State: Initialized, Role: FOLLOWER
I20260810 03:31:32.499248 28602 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6c2afa256e814479bd20f250d14ec80b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c2afa256e814479bd20f250d14ec80b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39505 } }
I20260810 03:31:32.499548 28602 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6c2afa256e814479bd20f250d14ec80b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 03:31:32.499713 28602 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6c2afa256e814479bd20f250d14ec80b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 03:31:32.499918 28602 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6c2afa256e814479bd20f250d14ec80b [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:31:32.503356 28602 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6c2afa256e814479bd20f250d14ec80b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c2afa256e814479bd20f250d14ec80b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39505 } }
I20260810 03:31:32.504098 28602 leader_election.cc:302] T 00000000000000000000000000000000 P 6c2afa256e814479bd20f250d14ec80b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6c2afa256e814479bd20f250d14ec80b; no voters:
I20260810 03:31:32.504895 28602 leader_election.cc:288] T 00000000000000000000000000000000 P 6c2afa256e814479bd20f250d14ec80b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 03:31:32.505108 28615 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6c2afa256e814479bd20f250d14ec80b [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:31:32.506079 28605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:31:32.497893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.27:38665 (local address 127.0.0.1:39505)
0810 03:31:32.498118 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0810 03:31:32.498125 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:31:32.498441 (+ 316us) server_negotiation.cc:407] Connection header received
0810 03:31:32.498657 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:31:32.498663 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:31:32.498750 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:31:32.498867 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:31:32.499443 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:31:32.500559 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:31:32.502539 (+ 1980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:31:32.503271 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:31:32.503405 (+ 134us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:31:32.504742 (+ 1337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:31:32.504753 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:31:32.504759 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:31:32.504788 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:31:32.504855 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:31:32.504881 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:31:32.505033 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0810 03:31:32.505434 (+ 401us) server_negotiation.cc:299] Negotiation successful
0810 03:31:32.505576 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260810 03:31:32.506356 28615 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6c2afa256e814479bd20f250d14ec80b [term 1 LEADER]: Becoming Leader. State: Replica: 6c2afa256e814479bd20f250d14ec80b, State: Running, Role: LEADER
I20260810 03:31:32.507007 28615 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6c2afa256e814479bd20f250d14ec80b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c2afa256e814479bd20f250d14ec80b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39505 } }
I20260810 03:31:32.507503 28602 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6c2afa256e814479bd20f250d14ec80b [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:31:32.510187 28616 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c2afa256e814479bd20f250d14ec80b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6c2afa256e814479bd20f250d14ec80b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c2afa256e814479bd20f250d14ec80b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39505 } } }
I20260810 03:31:32.510203 28619 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c2afa256e814479bd20f250d14ec80b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6c2afa256e814479bd20f250d14ec80b. Latest consensus state: current_term: 1 leader_uuid: "6c2afa256e814479bd20f250d14ec80b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c2afa256e814479bd20f250d14ec80b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39505 } } }
I20260810 03:31:32.510692 28616 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c2afa256e814479bd20f250d14ec80b [sys.catalog]: This master's current role is: LEADER
I20260810 03:31:32.510692 28619 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c2afa256e814479bd20f250d14ec80b [sys.catalog]: This master's current role is: LEADER
I20260810 03:31:32.512473 28625 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 03:31:32.518361 28625 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 03:31:32.528630 28625 catalog_manager.cc:1347] Generated new cluster ID: 99cb148612d94383acefab5c5bc00c0b
I20260810 03:31:32.528776 28625 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 03:31:32.538949 28625 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 03:31:32.540915 28625 catalog_manager.cc:1504] Loading token signing keys...
I20260810 03:31:32.551420 28625 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6c2afa256e814479bd20f250d14ec80b: Generated new TSK 0
I20260810 03:31:32.552682 28625 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 03:31:32.564383 28292 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iBH5/build/asan/bin/kudu pid 28496
W20260810 03:31:32.565685 28594 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.27:35743
W20260810 03:31:32.566601 28292 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task24iBH5/build/asan/bin/kudu pid 28496: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260810 03:31:32.566751 28292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iBH5/build/asan/bin/kudu with pid 28496
W20260810 03:31:32.592691 28477 connection.cc:535] client connection to 127.0.0.1:39505 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39505): Connection reset by peer (error 104)
2026-08-10T03:31:32Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (2187 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786332336319544-28292-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 03:31:32 dist-test-slave-vpvm krb5kdc[28646](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 03:31:32 dist-test-slave-vpvm krb5kdc[28646](info): set up 2 sockets
Aug 10 03:31:32 dist-test-slave-vpvm krb5kdc[28646](info): commencing operation
krb5kdc: starting...
W20260810 03:31:33.828361 28292 mini_kdc.cc:121] Time spent starting KDC: real 1.216s user 0.000s sys 0.058s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786332336319544-28292-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786332336319544-28292-0/krb5kdc/test-user.keytab.
Aug 10 03:31:34 dist-test-slave-vpvm krb5kdc[28646](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332694, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T03:31:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T03:31:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786332336319544-28292-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786332336319544-28292-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 03:31:34.192786 28292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iBH5/build/asan/bin/kudu
/tmp/dist-test-task24iBH5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786332336319544-28292-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786332336319544-28292-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786332336319544-28292-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786332336319544-28292-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33727
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786332336319544-28292-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40457
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33727
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786332336319544-28292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786332336319544-28292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786332336319544-28292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:31:34.559651 28667 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:31:34.559978 28667 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:31:34.560127 28667 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:31:34.567590 28667 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:31:34.567734 28667 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:31:34.567817 28667 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:31:34.567902 28667 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:31:34.567962 28667 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:31:34.578176 28667 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40457
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786332336319544-28292-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786332336319544-28292-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33727
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786332336319544-28292-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33727
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786332336319544-28292-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786332336319544-28292-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:31:34.579597 28667 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:31:34.580966 28667 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:31:34.593335 28674 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:31:34.593255 28675 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:31:34.593295 28678 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:31:34.594244 28667 server_base.cc:1034] running on GCE node
I20260810 03:31:34.595137 28667 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:31:34.596737 28667 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:31:34.598026 28667 hybrid_clock.cc:648] HybridClock initialized: now 1786332694597948 us; error 58 us; skew 500 ppm
I20260810 03:31:34.598428 28667 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:31:34 dist-test-slave-vpvm krb5kdc[28646](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332694, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:31:34.601869 28667 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 03:31:34.603583 28667 webserver.cc:460] Webserver started at http://127.0.0.1:35651/ using document root <none> and password file <none>
I20260810 03:31:34.604298 28667 fs_manager.cc:359] Metadata directory not provided
I20260810 03:31:34.604424 28667 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:31:34.604743 28667 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:31:34.607920 28667 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786332336319544-28292-0/minicluster-data/master-0/data/instance:
uuid: "47492faf01094cdfbdb971295af9be41"
format_stamp: "Formatted at 2026-08-10 03:31:34 on dist-test-slave-vpvm"
server_key: "5c824e7e39da3d1b646335ceb3dacb43"
server_key_iv: "e54a3cd21aee47d3310070016604aee8"
server_key_version: "encryptionkey@0"
I20260810 03:31:34.608716 28667 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786332336319544-28292-0/minicluster-data/master-0/wal/instance:
uuid: "47492faf01094cdfbdb971295af9be41"
format_stamp: "Formatted at 2026-08-10 03:31:34 on dist-test-slave-vpvm"
server_key: "5c824e7e39da3d1b646335ceb3dacb43"
server_key_iv: "e54a3cd21aee47d3310070016604aee8"
server_key_version: "encryptionkey@0"
I20260810 03:31:34.614343 28667 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260810 03:31:34.618364 28687 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:31:34.620106 28667 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260810 03:31:34.620316 28667 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786332336319544-28292-0/minicluster-data/master-0/data,/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786332336319544-28292-0/minicluster-data/master-0/wal
uuid: "47492faf01094cdfbdb971295af9be41"
format_stamp: "Formatted at 2026-08-10 03:31:34 on dist-test-slave-vpvm"
server_key: "5c824e7e39da3d1b646335ceb3dacb43"
server_key_iv: "e54a3cd21aee47d3310070016604aee8"
server_key_version: "encryptionkey@0"
I20260810 03:31:34.620536 28667 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786332336319544-28292-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786332336319544-28292-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786332336319544-28292-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:31:34.626971 28667 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:31:34.632004 28667 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:31:34.632337 28667 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:31:34.651605 28667 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33727
I20260810 03:31:34.651656 28763 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33727 every 8 connection(s)
I20260810 03:31:34.653115 28667 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786332336319544-28292-0/minicluster-data/master-0/data/info.pb
I20260810 03:31:34.657164 28764 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:31:34.660701 28292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iBH5/build/asan/bin/kudu as pid 28667
I20260810 03:31:34.660887 28292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786332336319544-28292-0/minicluster-data/master-0/wal/instance
I20260810 03:31:34.661316 28292 external_mini_cluster.cc:1428] Setting key 76a8645413f017314e491fe499f0e169
I20260810 03:31:34.670717 28767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:31:34.663360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.27:40621 (local address 127.0.0.1:33727)
0810 03:31:34.664261 (+ 901us) server_negotiation.cc:206] Beginning negotiation
0810 03:31:34.664278 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0810 03:31:34.664332 (+ 54us) server_negotiation.cc:407] Connection header received
0810 03:31:34.666440 (+ 2108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:31:34.666474 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:31:34.667015 (+ 541us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:31:34.667512 (+ 497us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:31:34.668574 (+ 1062us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:31:34.668582 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:31:34.668600 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:31:34.668623 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:31:34.668710 (+ 87us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:31:34.668739 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:31:34.668984 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0810 03:31:34.669170 (+ 186us) server_negotiation.cc:299] Negotiation successful
0810 03:31:34.669381 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":643,"thread_start_us":368,"threads_started":1}
I20260810 03:31:34.671288 28764 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 47492faf01094cdfbdb971295af9be41: Bootstrap starting.
I20260810 03:31:34.674656 28764 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 47492faf01094cdfbdb971295af9be41: Neither blocks nor log segments found. Creating new log.
I20260810 03:31:34.674772 28292 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 03:31:34.675990 28764 log.cc:824] T 00000000000000000000000000000000 P 47492faf01094cdfbdb971295af9be41: Log is configured to *not* fsync() on all Append() calls
I20260810 03:31:34.679750 28764 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 47492faf01094cdfbdb971295af9be41: No bootstrap required, opened a new log
I20260810 03:31:34.682812 28767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:31:34.680128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.27:40075 (local address 127.0.0.1:33727)
0810 03:31:34.680358 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0810 03:31:34.680365 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:31:34.680761 (+ 396us) server_negotiation.cc:407] Connection header received
0810 03:31:34.680946 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:31:34.680950 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:31:34.681002 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:31:34.681085 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:31:34.682142 (+ 1057us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:31:34.682150 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:31:34.682153 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:31:34.682169 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:31:34.682211 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:31:34.682226 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:31:34.682321 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0810 03:31:34.682443 (+ 122us) server_negotiation.cc:299] Negotiation successful
0810 03:31:34.682504 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260810 03:31:34.688653 28764 raft_consensus.cc:348] T 00000000000000000000000000000000 P 47492faf01094cdfbdb971295af9be41 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47492faf01094cdfbdb971295af9be41" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33727 } }
I20260810 03:31:34.689071 28764 raft_consensus.cc:374] T 00000000000000000000000000000000 P 47492faf01094cdfbdb971295af9be41 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:31:34.689163 28764 raft_consensus.cc:729] T 00000000000000000000000000000000 P 47492faf01094cdfbdb971295af9be41 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47492faf01094cdfbdb971295af9be41, State: Initialized, Role: FOLLOWER
I20260810 03:31:34.689744 28764 consensus_queue.cc:260] T 00000000000000000000000000000000 P 47492faf01094cdfbdb971295af9be41 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47492faf01094cdfbdb971295af9be41" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33727 } }
I20260810 03:31:34.690025 28764 raft_consensus.cc:388] T 00000000000000000000000000000000 P 47492faf01094cdfbdb971295af9be41 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 03:31:34.690164 28764 raft_consensus.cc:482] T 00000000000000000000000000000000 P 47492faf01094cdfbdb971295af9be41 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 03:31:34.690323 28764 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 47492faf01094cdfbdb971295af9be41 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:31:34.692693 28764 raft_consensus.cc:504] T 00000000000000000000000000000000 P 47492faf01094cdfbdb971295af9be41 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47492faf01094cdfbdb971295af9be41" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33727 } }
I20260810 03:31:34.693230 28764 leader_election.cc:302] T 00000000000000000000000000000000 P 47492faf01094cdfbdb971295af9be41 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 47492faf01094cdfbdb971295af9be41; no voters:
I20260810 03:31:34.693831 28764 leader_election.cc:288] T 00000000000000000000000000000000 P 47492faf01094cdfbdb971295af9be41 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 03:31:34.694010 28777 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 47492faf01094cdfbdb971295af9be41 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:31:34.694656 28777 raft_consensus.cc:686] T 00000000000000000000000000000000 P 47492faf01094cdfbdb971295af9be41 [term 1 LEADER]: Becoming Leader. State: Replica: 47492faf01094cdfbdb971295af9be41, State: Running, Role: LEADER
I20260810 03:31:34.695220 28777 consensus_queue.cc:237] T 00000000000000000000000000000000 P 47492faf01094cdfbdb971295af9be41 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47492faf01094cdfbdb971295af9be41" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33727 } }
I20260810 03:31:34.695933 28764 sys_catalog.cc:564] T 00000000000000000000000000000000 P 47492faf01094cdfbdb971295af9be41 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:31:34.697945 28779 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47492faf01094cdfbdb971295af9be41 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "47492faf01094cdfbdb971295af9be41" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47492faf01094cdfbdb971295af9be41" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33727 } } }
I20260810 03:31:34.698511 28779 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47492faf01094cdfbdb971295af9be41 [sys.catalog]: This master's current role is: LEADER
I20260810 03:31:34.698614 28780 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47492faf01094cdfbdb971295af9be41 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 47492faf01094cdfbdb971295af9be41. Latest consensus state: current_term: 1 leader_uuid: "47492faf01094cdfbdb971295af9be41" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47492faf01094cdfbdb971295af9be41" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33727 } } }
I20260810 03:31:34.698997 28780 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47492faf01094cdfbdb971295af9be41 [sys.catalog]: This master's current role is: LEADER
I20260810 03:31:34.700654 28791 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 03:31:34.706163 28791 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 03:31:34.716365 28791 catalog_manager.cc:1347] Generated new cluster ID: d1ba8504312e4693b3145313953a063c
I20260810 03:31:34.716480 28791 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 03:31:34.728250 28791 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 03:31:34.728688 28791 catalog_manager.cc:1504] Loading token signing keys...
I20260810 03:31:34.736919 28791 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 47492faf01094cdfbdb971295af9be41: Generated new TSK 0
I20260810 03:31:34.738116 28791 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 03:31:34.749936 28292 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iBH5/build/asan/bin/kudu pid 28667
W20260810 03:31:34.751190 28756 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.27:40621
W20260810 03:31:34.751896 28292 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task24iBH5/build/asan/bin/kudu pid 28667: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260810 03:31:34.752069 28292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iBH5/build/asan/bin/kudu with pid 28667
2026-08-10T03:31:34Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (2189 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786332336319544-28292-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 03:31:34 dist-test-slave-vpvm krb5kdc[28808](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 03:31:34 dist-test-slave-vpvm krb5kdc[28808](info): set up 2 sockets
Aug 10 03:31:34 dist-test-slave-vpvm krb5kdc[28808](info): commencing operation
krb5kdc: starting...
W20260810 03:31:36.024431 28292 mini_kdc.cc:121] Time spent starting KDC: real 1.223s user 0.004s sys 0.053s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786332336319544-28292-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786332336319544-28292-0/krb5kdc/test-user.keytab.
Aug 10 03:31:36 dist-test-slave-vpvm krb5kdc[28808](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332696, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T03:31:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T03:31:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786332336319544-28292-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786332336319544-28292-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 03:31:36.387677 28292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iBH5/build/asan/bin/kudu
/tmp/dist-test-task24iBH5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786332336319544-28292-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786332336319544-28292-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786332336319544-28292-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786332336319544-28292-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46385
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786332336319544-28292-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33173
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46385
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786332336319544-28292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786332336319544-28292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786332336319544-28292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:31:36.751719 28823 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:31:36.752070 28823 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:31:36.752215 28823 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:31:36.759963 28823 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:31:36.760119 28823 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:31:36.760190 28823 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:31:36.760288 28823 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:31:36.760365 28823 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:31:36.771203 28823 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33173
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786332336319544-28292-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786332336319544-28292-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46385
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786332336319544-28292-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46385
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786332336319544-28292-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786332336319544-28292-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:31:36.772660 28823 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:31:36.774111 28823 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:31:36.786811 28830 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:31:36.786746 28832 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:31:36.786823 28834 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:31:36.787935 28823 server_base.cc:1034] running on GCE node
I20260810 03:31:36.788766 28823 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:31:36.790359 28823 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:31:36.791618 28823 hybrid_clock.cc:648] HybridClock initialized: now 1786332696791563 us; error 44 us; skew 500 ppm
I20260810 03:31:36.792002 28823 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:31:36 dist-test-slave-vpvm krb5kdc[28808](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332696, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:31:36.795572 28823 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 03:31:36.797221 28823 webserver.cc:460] Webserver started at http://127.0.0.1:39845/ using document root <none> and password file <none>
I20260810 03:31:36.797995 28823 fs_manager.cc:359] Metadata directory not provided
I20260810 03:31:36.798131 28823 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:31:36.798420 28823 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:31:36.801635 28823 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786332336319544-28292-0/minicluster-data/master-0/data/instance:
uuid: "028c0de03fa34abeb2a5070f94fe6d6e"
format_stamp: "Formatted at 2026-08-10 03:31:36 on dist-test-slave-vpvm"
server_key: "c5c37b5aede3dc755576f3fcb4c29b2a"
server_key_iv: "949ec112dcb0c1b0219944eead419555"
server_key_version: "encryptionkey@0"
I20260810 03:31:36.802546 28823 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786332336319544-28292-0/minicluster-data/master-0/wal/instance:
uuid: "028c0de03fa34abeb2a5070f94fe6d6e"
format_stamp: "Formatted at 2026-08-10 03:31:36 on dist-test-slave-vpvm"
server_key: "c5c37b5aede3dc755576f3fcb4c29b2a"
server_key_iv: "949ec112dcb0c1b0219944eead419555"
server_key_version: "encryptionkey@0"
I20260810 03:31:36.808665 28823 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260810 03:31:36.812786 28842 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:31:36.814379 28823 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260810 03:31:36.814554 28823 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786332336319544-28292-0/minicluster-data/master-0/data,/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786332336319544-28292-0/minicluster-data/master-0/wal
uuid: "028c0de03fa34abeb2a5070f94fe6d6e"
format_stamp: "Formatted at 2026-08-10 03:31:36 on dist-test-slave-vpvm"
server_key: "c5c37b5aede3dc755576f3fcb4c29b2a"
server_key_iv: "949ec112dcb0c1b0219944eead419555"
server_key_version: "encryptionkey@0"
I20260810 03:31:36.814741 28823 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786332336319544-28292-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786332336319544-28292-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786332336319544-28292-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:31:36.832688 28823 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:31:36.836822 28823 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:31:36.837081 28823 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:31:36.855532 28823 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46385
I20260810 03:31:36.855556 28915 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46385 every 8 connection(s)
I20260810 03:31:36.856957 28823 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786332336319544-28292-0/minicluster-data/master-0/data/info.pb
I20260810 03:31:36.860766 28916 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:31:36.863963 28292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iBH5/build/asan/bin/kudu as pid 28823
I20260810 03:31:36.864123 28292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786332336319544-28292-0/minicluster-data/master-0/wal/instance
I20260810 03:31:36.864557 28292 external_mini_cluster.cc:1428] Setting key efe95170c7c9f65f7f5cd9d69ee8b100
I20260810 03:31:36.873983 28916 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 028c0de03fa34abeb2a5070f94fe6d6e: Bootstrap starting.
I20260810 03:31:36.877787 28916 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 028c0de03fa34abeb2a5070f94fe6d6e: Neither blocks nor log segments found. Creating new log.
I20260810 03:31:36.878854 28916 log.cc:824] T 00000000000000000000000000000000 P 028c0de03fa34abeb2a5070f94fe6d6e: Log is configured to *not* fsync() on all Append() calls
Aug 10 03:31:36 dist-test-slave-vpvm krb5kdc[28808](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332696, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260810 03:31:36.882115 28916 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 028c0de03fa34abeb2a5070f94fe6d6e: No bootstrap required, opened a new log
I20260810 03:31:36.889496 28922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:31:36.866542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.27:47957 (local address 127.0.0.1:46385)
0810 03:31:36.867303 (+ 761us) server_negotiation.cc:206] Beginning negotiation
0810 03:31:36.867319 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 03:31:36.867371 (+ 52us) server_negotiation.cc:407] Connection header received
0810 03:31:36.869593 (+ 2222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:31:36.869635 (+ 42us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:31:36.870258 (+ 623us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:31:36.870720 (+ 462us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:31:36.872117 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:31:36.874151 (+ 2034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:31:36.875461 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:31:36.876508 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:31:36.876713 (+ 205us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:31:36.881292 (+ 4579us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:31:36.881301 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:31:36.881316 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:31:36.881357 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:31:36.884570 (+ 3213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:31:36.885654 (+ 1084us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:31:36.885662 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:31:36.885669 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:31:36.885880 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:31:36.886425 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:31:36.886429 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:31:36.886431 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:31:36.887302 (+ 871us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:31:36.887666 (+ 364us) server_negotiation.cc:1035] Waiting for connection context
0810 03:31:36.888112 (+ 446us) server_negotiation.cc:299] Negotiation successful
0810 03:31:36.888397 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":521,"thread_start_us":297,"threads_started":1}
I20260810 03:31:36.891397 28916 raft_consensus.cc:348] T 00000000000000000000000000000000 P 028c0de03fa34abeb2a5070f94fe6d6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "028c0de03fa34abeb2a5070f94fe6d6e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46385 } }
I20260810 03:31:36.891927 28916 raft_consensus.cc:374] T 00000000000000000000000000000000 P 028c0de03fa34abeb2a5070f94fe6d6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:31:36.892055 28916 raft_consensus.cc:729] T 00000000000000000000000000000000 P 028c0de03fa34abeb2a5070f94fe6d6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 028c0de03fa34abeb2a5070f94fe6d6e, State: Initialized, Role: FOLLOWER
I20260810 03:31:36.892674 28916 consensus_queue.cc:260] T 00000000000000000000000000000000 P 028c0de03fa34abeb2a5070f94fe6d6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "028c0de03fa34abeb2a5070f94fe6d6e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46385 } }
I20260810 03:31:36.892967 28916 raft_consensus.cc:388] T 00000000000000000000000000000000 P 028c0de03fa34abeb2a5070f94fe6d6e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 03:31:36.893117 28916 raft_consensus.cc:482] T 00000000000000000000000000000000 P 028c0de03fa34abeb2a5070f94fe6d6e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 03:31:36.893292 28916 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 028c0de03fa34abeb2a5070f94fe6d6e [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:31:36.893748 28292 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 03:31:36.895722 28916 raft_consensus.cc:504] T 00000000000000000000000000000000 P 028c0de03fa34abeb2a5070f94fe6d6e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "028c0de03fa34abeb2a5070f94fe6d6e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46385 } }
I20260810 03:31:36.896286 28916 leader_election.cc:302] T 00000000000000000000000000000000 P 028c0de03fa34abeb2a5070f94fe6d6e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 028c0de03fa34abeb2a5070f94fe6d6e; no voters:
I20260810 03:31:36.896993 28916 leader_election.cc:288] T 00000000000000000000000000000000 P 028c0de03fa34abeb2a5070f94fe6d6e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 03:31:36.897097 28930 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 028c0de03fa34abeb2a5070f94fe6d6e [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:31:36.898239 28930 raft_consensus.cc:686] T 00000000000000000000000000000000 P 028c0de03fa34abeb2a5070f94fe6d6e [term 1 LEADER]: Becoming Leader. State: Replica: 028c0de03fa34abeb2a5070f94fe6d6e, State: Running, Role: LEADER
I20260810 03:31:36.898918 28930 consensus_queue.cc:237] T 00000000000000000000000000000000 P 028c0de03fa34abeb2a5070f94fe6d6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "028c0de03fa34abeb2a5070f94fe6d6e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46385 } }
I20260810 03:31:36.899592 28916 sys_catalog.cc:564] T 00000000000000000000000000000000 P 028c0de03fa34abeb2a5070f94fe6d6e [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:31:36.902101 28933 sys_catalog.cc:455] T 00000000000000000000000000000000 P 028c0de03fa34abeb2a5070f94fe6d6e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 028c0de03fa34abeb2a5070f94fe6d6e. Latest consensus state: current_term: 1 leader_uuid: "028c0de03fa34abeb2a5070f94fe6d6e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "028c0de03fa34abeb2a5070f94fe6d6e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46385 } } }
I20260810 03:31:36.902524 28933 sys_catalog.cc:458] T 00000000000000000000000000000000 P 028c0de03fa34abeb2a5070f94fe6d6e [sys.catalog]: This master's current role is: LEADER
I20260810 03:31:36.907409 28932 sys_catalog.cc:455] T 00000000000000000000000000000000 P 028c0de03fa34abeb2a5070f94fe6d6e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "028c0de03fa34abeb2a5070f94fe6d6e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "028c0de03fa34abeb2a5070f94fe6d6e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46385 } } }
I20260810 03:31:36.908377 28932 sys_catalog.cc:458] T 00000000000000000000000000000000 P 028c0de03fa34abeb2a5070f94fe6d6e [sys.catalog]: This master's current role is: LEADER
I20260810 03:31:36.911231 28937 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 03:31:36.917860 28937 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 03:31:36.919188 28922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:31:36.898775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.27:37365 (local address 127.0.0.1:46385)
0810 03:31:36.898964 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0810 03:31:36.898970 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:31:36.899141 (+ 171us) server_negotiation.cc:407] Connection header received
0810 03:31:36.899382 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:31:36.899387 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:31:36.899463 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:31:36.899688 (+ 225us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:31:36.901570 (+ 1882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:31:36.902768 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:31:36.904783 (+ 2015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:31:36.905527 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:31:36.905635 (+ 108us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:31:36.909648 (+ 4013us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:31:36.909666 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:31:36.909680 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:31:36.909736 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:31:36.914196 (+ 4460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:31:36.915375 (+ 1179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:31:36.915381 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:31:36.915385 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:31:36.915547 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:31:36.916152 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:31:36.916157 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:31:36.916164 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:31:36.916656 (+ 492us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:31:36.916940 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0810 03:31:36.918407 (+ 1467us) server_negotiation.cc:299] Negotiation successful
0810 03:31:36.918607 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260810 03:31:36.928608 28937 catalog_manager.cc:1347] Generated new cluster ID: be24836058194df3ae06fc2f54c0bd33
I20260810 03:31:36.928774 28937 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 03:31:36.943560 28937 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 03:31:36.944887 28937 catalog_manager.cc:1504] Loading token signing keys...
I20260810 03:31:36.962564 28937 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 028c0de03fa34abeb2a5070f94fe6d6e: Generated new TSK 0
I20260810 03:31:36.963691 28937 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 03:31:36.978334 28292 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iBH5/build/asan/bin/kudu pid 28823
I20260810 03:31:36.979892 28908 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:31:37.074755 28292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iBH5/build/asan/bin/kudu with pid 28823
W20260810 03:31:37.099874 28806 connection.cc:535] client connection to 127.0.0.1:46385 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46385): Connection reset by peer (error 104)
2026-08-10T03:31:37Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (2318 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786332336319544-28292-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 03:31:37 dist-test-slave-vpvm krb5kdc[28962](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 03:31:37 dist-test-slave-vpvm krb5kdc[28962](info): set up 2 sockets
Aug 10 03:31:37 dist-test-slave-vpvm krb5kdc[28962](info): commencing operation
krb5kdc: starting...
W20260810 03:31:38.341274 28292 mini_kdc.cc:121] Time spent starting KDC: real 1.221s user 0.003s sys 0.056s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786332336319544-28292-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786332336319544-28292-0/krb5kdc/test-user.keytab.
Aug 10 03:31:38 dist-test-slave-vpvm krb5kdc[28962](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332698, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T03:31:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T03:31:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786332336319544-28292-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786332336319544-28292-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 03:31:38.703670 28292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iBH5/build/asan/bin/kudu
/tmp/dist-test-task24iBH5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786332336319544-28292-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786332336319544-28292-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786332336319544-28292-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786332336319544-28292-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45737
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786332336319544-28292-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45737
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786332336319544-28292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786332336319544-28292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786332336319544-28292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:31:39.065843 28982 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:31:39.066315 28982 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:31:39.066494 28982 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:31:39.074343 28982 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:31:39.074513 28982 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:31:39.074605 28982 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:31:39.074705 28982 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:31:39.074779 28982 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:31:39.085498 28982 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786332336319544-28292-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786332336319544-28292-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45737
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786332336319544-28292-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45737
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786332336319544-28292-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786332336319544-28292-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:31:39.087000 28982 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:31:39.088425 28982 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:31:39.100987 28987 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:31:39.101032 28988 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:31:39.101089 28991 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:31:39.102209 28982 server_base.cc:1034] running on GCE node
I20260810 03:31:39.103130 28982 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:31:39.104781 28982 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:31:39.106078 28982 hybrid_clock.cc:648] HybridClock initialized: now 1786332699106020 us; error 42 us; skew 500 ppm
I20260810 03:31:39.106452 28982 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:31:39 dist-test-slave-vpvm krb5kdc[28962](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332699, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:31:39.109644 28982 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 03:31:39.111246 28982 webserver.cc:460] Webserver started at http://127.0.0.1:42117/ using document root <none> and password file <none>
I20260810 03:31:39.111970 28982 fs_manager.cc:359] Metadata directory not provided
I20260810 03:31:39.112128 28982 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:31:39.112442 28982 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:31:39.115717 28982 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786332336319544-28292-0/minicluster-data/master-0/data/instance:
uuid: "dba4ab7422df47e2a6320fd8abee5061"
format_stamp: "Formatted at 2026-08-10 03:31:39 on dist-test-slave-vpvm"
server_key: "18e28ea96ddadaac07cf88ca7c39aa72"
server_key_iv: "83fe9e6feb99ba7ce4d877383579c203"
server_key_version: "encryptionkey@0"
I20260810 03:31:39.116523 28982 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786332336319544-28292-0/minicluster-data/master-0/wal/instance:
uuid: "dba4ab7422df47e2a6320fd8abee5061"
format_stamp: "Formatted at 2026-08-10 03:31:39 on dist-test-slave-vpvm"
server_key: "18e28ea96ddadaac07cf88ca7c39aa72"
server_key_iv: "83fe9e6feb99ba7ce4d877383579c203"
server_key_version: "encryptionkey@0"
I20260810 03:31:39.122254 28982 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.007s
I20260810 03:31:39.126042 28999 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:31:39.127449 28982 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 03:31:39.127631 28982 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786332336319544-28292-0/minicluster-data/master-0/data,/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786332336319544-28292-0/minicluster-data/master-0/wal
uuid: "dba4ab7422df47e2a6320fd8abee5061"
format_stamp: "Formatted at 2026-08-10 03:31:39 on dist-test-slave-vpvm"
server_key: "18e28ea96ddadaac07cf88ca7c39aa72"
server_key_iv: "83fe9e6feb99ba7ce4d877383579c203"
server_key_version: "encryptionkey@0"
I20260810 03:31:39.127835 28982 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786332336319544-28292-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786332336319544-28292-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786332336319544-28292-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:31:39.146452 28982 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:31:39.150673 28982 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:31:39.150942 28982 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:31:39.168578 28982 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45737
I20260810 03:31:39.168601 29068 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45737 every 8 connection(s)
I20260810 03:31:39.170055 28982 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786332336319544-28292-0/minicluster-data/master-0/data/info.pb
I20260810 03:31:39.170435 28292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iBH5/build/asan/bin/kudu as pid 28982
I20260810 03:31:39.170583 28292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786332336319544-28292-0/minicluster-data/master-0/wal/instance
I20260810 03:31:39.171070 28292 external_mini_cluster.cc:1428] Setting key 32c8a48347f0f0862de5a2e056138058
I20260810 03:31:39.174396 29069 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:31:39.184202 29074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:31:39.172907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.27:59117 (local address 127.0.0.1:45737)
0810 03:31:39.173678 (+ 771us) server_negotiation.cc:206] Beginning negotiation
0810 03:31:39.173689 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 03:31:39.173729 (+ 40us) server_negotiation.cc:407] Connection header received
0810 03:31:39.175439 (+ 1710us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:31:39.175463 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:31:39.175838 (+ 375us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:31:39.176133 (+ 295us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:31:39.176708 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:31:39.178065 (+ 1357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:31:39.179842 (+ 1777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:31:39.180506 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:31:39.180640 (+ 134us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:31:39.181886 (+ 1246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:31:39.181894 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:31:39.181908 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:31:39.182461 (+ 553us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:31:39.182549 (+ 88us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:31:39.182577 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:31:39.182708 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0810 03:31:39.182958 (+ 250us) server_negotiation.cc:299] Negotiation successful
0810 03:31:39.183144 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":548,"thread_start_us":328,"threads_started":1}
I20260810 03:31:39.186725 29069 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dba4ab7422df47e2a6320fd8abee5061: Bootstrap starting.
I20260810 03:31:39.188241 28292 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 03:31:39.190141 29069 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dba4ab7422df47e2a6320fd8abee5061: Neither blocks nor log segments found. Creating new log.
I20260810 03:31:39.191303 29069 log.cc:824] T 00000000000000000000000000000000 P dba4ab7422df47e2a6320fd8abee5061: Log is configured to *not* fsync() on all Append() calls
I20260810 03:31:39.194500 29069 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dba4ab7422df47e2a6320fd8abee5061: No bootstrap required, opened a new log
I20260810 03:31:39.198860 29074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:31:39.192596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.27:43297 (local address 127.0.0.1:45737)
0810 03:31:39.192751 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0810 03:31:39.192756 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:31:39.193199 (+ 443us) server_negotiation.cc:407] Connection header received
0810 03:31:39.193407 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:31:39.193412 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:31:39.193476 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:31:39.193571 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:31:39.194142 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:31:39.194959 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:31:39.196298 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:31:39.196780 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:31:39.196902 (+ 122us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:31:39.198118 (+ 1216us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:31:39.198122 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:31:39.198126 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:31:39.198145 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:31:39.198184 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:31:39.198199 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:31:39.198293 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0810 03:31:39.198496 (+ 203us) server_negotiation.cc:299] Negotiation successful
0810 03:31:39.198562 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260810 03:31:39.202777 29069 raft_consensus.cc:348] T 00000000000000000000000000000000 P dba4ab7422df47e2a6320fd8abee5061 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dba4ab7422df47e2a6320fd8abee5061" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45737 } }
I20260810 03:31:39.203189 29069 raft_consensus.cc:374] T 00000000000000000000000000000000 P dba4ab7422df47e2a6320fd8abee5061 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:31:39.203298 29069 raft_consensus.cc:729] T 00000000000000000000000000000000 P dba4ab7422df47e2a6320fd8abee5061 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dba4ab7422df47e2a6320fd8abee5061, State: Initialized, Role: FOLLOWER
I20260810 03:31:39.203939 29069 consensus_queue.cc:260] T 00000000000000000000000000000000 P dba4ab7422df47e2a6320fd8abee5061 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dba4ab7422df47e2a6320fd8abee5061" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45737 } }
I20260810 03:31:39.204185 29069 raft_consensus.cc:388] T 00000000000000000000000000000000 P dba4ab7422df47e2a6320fd8abee5061 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 03:31:39.204320 29069 raft_consensus.cc:482] T 00000000000000000000000000000000 P dba4ab7422df47e2a6320fd8abee5061 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 03:31:39.204507 29069 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dba4ab7422df47e2a6320fd8abee5061 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:31:39.206881 29069 raft_consensus.cc:504] T 00000000000000000000000000000000 P dba4ab7422df47e2a6320fd8abee5061 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dba4ab7422df47e2a6320fd8abee5061" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45737 } }
I20260810 03:31:39.207365 29069 leader_election.cc:302] T 00000000000000000000000000000000 P dba4ab7422df47e2a6320fd8abee5061 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dba4ab7422df47e2a6320fd8abee5061; no voters:
I20260810 03:31:39.207959 29069 leader_election.cc:288] T 00000000000000000000000000000000 P dba4ab7422df47e2a6320fd8abee5061 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 03:31:39.208105 29084 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dba4ab7422df47e2a6320fd8abee5061 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:31:39.210007 29069 sys_catalog.cc:564] T 00000000000000000000000000000000 P dba4ab7422df47e2a6320fd8abee5061 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:31:39.210405 29084 raft_consensus.cc:686] T 00000000000000000000000000000000 P dba4ab7422df47e2a6320fd8abee5061 [term 1 LEADER]: Becoming Leader. State: Replica: dba4ab7422df47e2a6320fd8abee5061, State: Running, Role: LEADER
I20260810 03:31:39.210963 29084 consensus_queue.cc:237] T 00000000000000000000000000000000 P dba4ab7422df47e2a6320fd8abee5061 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dba4ab7422df47e2a6320fd8abee5061" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45737 } }
I20260810 03:31:39.214010 29088 sys_catalog.cc:455] T 00000000000000000000000000000000 P dba4ab7422df47e2a6320fd8abee5061 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dba4ab7422df47e2a6320fd8abee5061. Latest consensus state: current_term: 1 leader_uuid: "dba4ab7422df47e2a6320fd8abee5061" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dba4ab7422df47e2a6320fd8abee5061" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45737 } } }
I20260810 03:31:39.214010 29087 sys_catalog.cc:455] T 00000000000000000000000000000000 P dba4ab7422df47e2a6320fd8abee5061 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dba4ab7422df47e2a6320fd8abee5061" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dba4ab7422df47e2a6320fd8abee5061" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45737 } } }
I20260810 03:31:39.214524 29087 sys_catalog.cc:458] T 00000000000000000000000000000000 P dba4ab7422df47e2a6320fd8abee5061 [sys.catalog]: This master's current role is: LEADER
I20260810 03:31:39.214524 29088 sys_catalog.cc:458] T 00000000000000000000000000000000 P dba4ab7422df47e2a6320fd8abee5061 [sys.catalog]: This master's current role is: LEADER
I20260810 03:31:39.215327 29097 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 03:31:39.221335 29097 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 03:31:39.231355 29097 catalog_manager.cc:1347] Generated new cluster ID: 3e5115cd3a244f94a26fef465edcd9cd
I20260810 03:31:39.231473 29097 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 03:31:39.248726 29097 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 03:31:39.250171 29097 catalog_manager.cc:1504] Loading token signing keys...
I20260810 03:31:39.259505 29097 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dba4ab7422df47e2a6320fd8abee5061: Generated new TSK 0
I20260810 03:31:39.260459 29097 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 03:31:39.279983 28292 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iBH5/build/asan/bin/kudu pid 28982
W20260810 03:31:39.281266 29063 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.27:59117
W20260810 03:31:39.282298 28292 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task24iBH5/build/asan/bin/kudu pid 28982: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260810 03:31:39.282500 28292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iBH5/build/asan/bin/kudu with pid 28982
W20260810 03:31:39.307369 28960 connection.cc:535] client connection to 127.0.0.1:45737 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45737): Connection reset by peer (error 104)
2026-08-10T03:31:39Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (2206 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786332336319544-28292-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 03:31:39 dist-test-slave-vpvm krb5kdc[29113](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 03:31:39 dist-test-slave-vpvm krb5kdc[29113](info): set up 2 sockets
Aug 10 03:31:39 dist-test-slave-vpvm krb5kdc[29113](info): commencing operation
krb5kdc: starting...
W20260810 03:31:40.536851 28292 mini_kdc.cc:121] Time spent starting KDC: real 1.210s user 0.001s sys 0.056s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786332336319544-28292-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786332336319544-28292-0/krb5kdc/test-user.keytab.
Aug 10 03:31:40 dist-test-slave-vpvm krb5kdc[29113](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332700, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T03:31:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T03:31:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786332336319544-28292-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786332336319544-28292-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 03:31:40.893345 28292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iBH5/build/asan/bin/kudu
/tmp/dist-test-task24iBH5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786332336319544-28292-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786332336319544-28292-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786332336319544-28292-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786332336319544-28292-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39959
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786332336319544-28292-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39959
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786332336319544-28292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786332336319544-28292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786332336319544-28292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:31:41.249565 29137 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:31:41.249891 29137 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:31:41.250056 29137 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:31:41.257367 29137 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:31:41.257507 29137 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:31:41.257591 29137 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:31:41.257673 29137 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:31:41.257737 29137 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:31:41.267867 29137 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786332336319544-28292-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786332336319544-28292-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39959
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786332336319544-28292-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39959
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786332336319544-28292-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786332336319544-28292-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:31:41.269258 29137 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:31:41.270713 29137 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:31:41.282490 29144 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:31:41.282531 29145 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:31:41.282744 29147 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:31:41.283509 29137 server_base.cc:1034] running on GCE node
I20260810 03:31:41.284403 29137 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:31:41.286000 29137 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:31:41.287266 29137 hybrid_clock.cc:648] HybridClock initialized: now 1786332701287207 us; error 53 us; skew 500 ppm
I20260810 03:31:41.287695 29137 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:31:41 dist-test-slave-vpvm krb5kdc[29113](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332701, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:31:41.294258 29137 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 03:31:41.295874 29137 webserver.cc:460] Webserver started at http://127.0.0.1:33901/ using document root <none> and password file <none>
I20260810 03:31:41.296631 29137 fs_manager.cc:359] Metadata directory not provided
I20260810 03:31:41.296759 29137 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:31:41.297070 29137 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:31:41.300307 29137 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786332336319544-28292-0/minicluster-data/master-0/data/instance:
uuid: "fd3cc7c4cab64a93b8c18542ee3323f0"
format_stamp: "Formatted at 2026-08-10 03:31:41 on dist-test-slave-vpvm"
server_key: "434f1fcb440d2aadd0c22e5e16524f3f"
server_key_iv: "06533847f71d56ebdadb34435f7df99d"
server_key_version: "encryptionkey@0"
I20260810 03:31:41.301122 29137 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786332336319544-28292-0/minicluster-data/master-0/wal/instance:
uuid: "fd3cc7c4cab64a93b8c18542ee3323f0"
format_stamp: "Formatted at 2026-08-10 03:31:41 on dist-test-slave-vpvm"
server_key: "434f1fcb440d2aadd0c22e5e16524f3f"
server_key_iv: "06533847f71d56ebdadb34435f7df99d"
server_key_version: "encryptionkey@0"
I20260810 03:31:41.306735 29137 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260810 03:31:41.310513 29158 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:31:41.311992 29137 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260810 03:31:41.312250 29137 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786332336319544-28292-0/minicluster-data/master-0/data,/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786332336319544-28292-0/minicluster-data/master-0/wal
uuid: "fd3cc7c4cab64a93b8c18542ee3323f0"
format_stamp: "Formatted at 2026-08-10 03:31:41 on dist-test-slave-vpvm"
server_key: "434f1fcb440d2aadd0c22e5e16524f3f"
server_key_iv: "06533847f71d56ebdadb34435f7df99d"
server_key_version: "encryptionkey@0"
I20260810 03:31:41.312470 29137 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786332336319544-28292-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786332336319544-28292-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786332336319544-28292-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:31:41.318709 29137 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:31:41.323256 29137 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:31:41.323593 29137 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:31:41.342022 29137 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39959
I20260810 03:31:41.342075 29246 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39959 every 8 connection(s)
I20260810 03:31:41.343494 29137 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786332336319544-28292-0/minicluster-data/master-0/data/info.pb
I20260810 03:31:41.347299 29248 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:31:41.349498 28292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iBH5/build/asan/bin/kudu as pid 29137
I20260810 03:31:41.349653 28292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786332336319544-28292-0/minicluster-data/master-0/wal/instance
I20260810 03:31:41.350114 28292 external_mini_cluster.cc:1428] Setting key 696535e16e270087fae804743c786515
I20260810 03:31:41.359393 29252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:31:41.352009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.27:43323 (local address 127.0.0.1:39959)
0810 03:31:41.352852 (+ 843us) server_negotiation.cc:206] Beginning negotiation
0810 03:31:41.352867 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 03:31:41.352925 (+ 58us) server_negotiation.cc:407] Connection header received
0810 03:31:41.355293 (+ 2368us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:31:41.355325 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:31:41.355791 (+ 466us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:31:41.356253 (+ 462us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:31:41.357274 (+ 1021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:31:41.357283 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:31:41.357300 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:31:41.357318 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:31:41.357391 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:31:41.357419 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:31:41.357674 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0810 03:31:41.357816 (+ 142us) server_negotiation.cc:299] Negotiation successful
0810 03:31:41.358060 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":590,"thread_start_us":361,"threads_started":1}
I20260810 03:31:41.360926 29248 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd3cc7c4cab64a93b8c18542ee3323f0: Bootstrap starting.
I20260810 03:31:41.363055 28292 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 03:31:41.364301 29248 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fd3cc7c4cab64a93b8c18542ee3323f0: Neither blocks nor log segments found. Creating new log.
I20260810 03:31:41.365331 29248 log.cc:824] T 00000000000000000000000000000000 P fd3cc7c4cab64a93b8c18542ee3323f0: Log is configured to *not* fsync() on all Append() calls
I20260810 03:31:41.368468 29248 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd3cc7c4cab64a93b8c18542ee3323f0: No bootstrap required, opened a new log
I20260810 03:31:41.369640 29252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:31:41.367374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.27:55339 (local address 127.0.0.1:39959)
0810 03:31:41.367569 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0810 03:31:41.367575 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:31:41.367866 (+ 291us) server_negotiation.cc:407] Connection header received
0810 03:31:41.368012 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:31:41.368017 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:31:41.368079 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:31:41.368193 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:31:41.369037 (+ 844us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:31:41.369043 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:31:41.369047 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:31:41.369066 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:31:41.369113 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:31:41.369135 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:31:41.369260 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0810 03:31:41.369307 (+ 47us) server_negotiation.cc:299] Negotiation successful
0810 03:31:41.369365 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260810 03:31:41.376533 29248 raft_consensus.cc:348] T 00000000000000000000000000000000 P fd3cc7c4cab64a93b8c18542ee3323f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd3cc7c4cab64a93b8c18542ee3323f0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39959 } }
I20260810 03:31:41.376914 29248 raft_consensus.cc:374] T 00000000000000000000000000000000 P fd3cc7c4cab64a93b8c18542ee3323f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:31:41.376993 29248 raft_consensus.cc:729] T 00000000000000000000000000000000 P fd3cc7c4cab64a93b8c18542ee3323f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd3cc7c4cab64a93b8c18542ee3323f0, State: Initialized, Role: FOLLOWER
I20260810 03:31:41.377503 29248 consensus_queue.cc:260] T 00000000000000000000000000000000 P fd3cc7c4cab64a93b8c18542ee3323f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd3cc7c4cab64a93b8c18542ee3323f0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39959 } }
I20260810 03:31:41.377732 29248 raft_consensus.cc:388] T 00000000000000000000000000000000 P fd3cc7c4cab64a93b8c18542ee3323f0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 03:31:41.377839 29248 raft_consensus.cc:482] T 00000000000000000000000000000000 P fd3cc7c4cab64a93b8c18542ee3323f0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 03:31:41.377995 29248 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fd3cc7c4cab64a93b8c18542ee3323f0 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:31:41.380448 29248 raft_consensus.cc:504] T 00000000000000000000000000000000 P fd3cc7c4cab64a93b8c18542ee3323f0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd3cc7c4cab64a93b8c18542ee3323f0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39959 } }
I20260810 03:31:41.380931 29248 leader_election.cc:302] T 00000000000000000000000000000000 P fd3cc7c4cab64a93b8c18542ee3323f0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fd3cc7c4cab64a93b8c18542ee3323f0; no voters:
I20260810 03:31:41.381453 29248 leader_election.cc:288] T 00000000000000000000000000000000 P fd3cc7c4cab64a93b8c18542ee3323f0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 03:31:41.381609 29261 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fd3cc7c4cab64a93b8c18542ee3323f0 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:31:41.382249 29261 raft_consensus.cc:686] T 00000000000000000000000000000000 P fd3cc7c4cab64a93b8c18542ee3323f0 [term 1 LEADER]: Becoming Leader. State: Replica: fd3cc7c4cab64a93b8c18542ee3323f0, State: Running, Role: LEADER
I20260810 03:31:41.382750 29261 consensus_queue.cc:237] T 00000000000000000000000000000000 P fd3cc7c4cab64a93b8c18542ee3323f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd3cc7c4cab64a93b8c18542ee3323f0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39959 } }
I20260810 03:31:41.383512 29248 sys_catalog.cc:564] T 00000000000000000000000000000000 P fd3cc7c4cab64a93b8c18542ee3323f0 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:31:41.385428 29265 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd3cc7c4cab64a93b8c18542ee3323f0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fd3cc7c4cab64a93b8c18542ee3323f0. Latest consensus state: current_term: 1 leader_uuid: "fd3cc7c4cab64a93b8c18542ee3323f0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd3cc7c4cab64a93b8c18542ee3323f0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39959 } } }
I20260810 03:31:41.385522 29263 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd3cc7c4cab64a93b8c18542ee3323f0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fd3cc7c4cab64a93b8c18542ee3323f0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd3cc7c4cab64a93b8c18542ee3323f0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39959 } } }
I20260810 03:31:41.385842 29265 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd3cc7c4cab64a93b8c18542ee3323f0 [sys.catalog]: This master's current role is: LEADER
I20260810 03:31:41.385867 29263 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd3cc7c4cab64a93b8c18542ee3323f0 [sys.catalog]: This master's current role is: LEADER
I20260810 03:31:41.387996 29269 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 03:31:41.393450 29269 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 03:31:41.402359 29269 catalog_manager.cc:1347] Generated new cluster ID: b7e59316b53b43cca4b130d6e5f598af
I20260810 03:31:41.402485 29269 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 03:31:41.437920 29269 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 03:31:41.438403 29269 catalog_manager.cc:1504] Loading token signing keys...
I20260810 03:31:41.446285 29269 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fd3cc7c4cab64a93b8c18542ee3323f0: Generated new TSK 0
I20260810 03:31:41.447420 29269 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 03:31:41.465854 28292 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iBH5/build/asan/bin/kudu pid 29137
W20260810 03:31:41.467234 29241 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.27:43323
W20260810 03:31:41.468006 28292 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task24iBH5/build/asan/bin/kudu pid 29137: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260810 03:31:41.468151 28292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iBH5/build/asan/bin/kudu with pid 29137
2026-08-10T03:31:41Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (2190 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786332336319544-28292-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 03:31:41 dist-test-slave-vpvm krb5kdc[29289](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 03:31:41 dist-test-slave-vpvm krb5kdc[29289](info): set up 2 sockets
Aug 10 03:31:41 dist-test-slave-vpvm krb5kdc[29289](info): commencing operation
krb5kdc: starting...
W20260810 03:31:42.750239 28292 mini_kdc.cc:121] Time spent starting KDC: real 1.233s user 0.000s sys 0.060s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786332336319544-28292-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786332336319544-28292-0/krb5kdc/test-user.keytab.
Aug 10 03:31:42 dist-test-slave-vpvm krb5kdc[29289](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332702, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T03:31:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T03:31:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.161.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.161.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.161.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.62.keytab.
Entry for principal kudu/127.27.161.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.62.keytab.
I20260810 03:31:43.111789 28292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iBH5/build/asan/bin/kudu
/tmp/dist-test-task24iBH5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786332336319544-28292-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786332336319544-28292-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786332336319544-28292-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786332336319544-28292-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.161.62:34437
--webserver_interface=127.27.161.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.62.keytab
--principal=kudu/127.27.161.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.161.20:41309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.161.62:34437
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786332336319544-28292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786332336319544-28292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786332336319544-28292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:31:43.473152 29307 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:31:43.473479 29307 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:31:43.473621 29307 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:31:43.481144 29307 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:31:43.481287 29307 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:31:43.481370 29307 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:31:43.481449 29307 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:31:43.481508 29307 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:31:43.491886 29307 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.161.20:41309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786332336319544-28292-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786332336319544-28292-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.161.62:34437
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.62.keytab
--principal=kudu/127.27.161.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.161.62:34437
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786332336319544-28292-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.161.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786332336319544-28292-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:31:43.493291 29307 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:31:43.494715 29307 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:31:43.506556 29316 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:31:43.506516 29320 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:31:43.507025 29315 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:31:43.507691 29307 server_base.cc:1034] running on GCE node
I20260810 03:31:43.508453 29307 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:31:43.509871 29307 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:31:43.511106 29307 hybrid_clock.cc:648] HybridClock initialized: now 1786332703511058 us; error 34 us; skew 500 ppm
I20260810 03:31:43.511466 29307 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:31:43 dist-test-slave-vpvm krb5kdc[29289](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332703, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.161.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:31:43.514441 29307 init.cc:377] Logged in from keytab as kudu/127.27.161.62@KRBTEST.COM (short username kudu)
I20260810 03:31:43.515942 29307 webserver.cc:460] Webserver started at http://127.27.161.62:35945/ using document root <none> and password file <none>
I20260810 03:31:43.516642 29307 fs_manager.cc:359] Metadata directory not provided
I20260810 03:31:43.516778 29307 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:31:43.517093 29307 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:31:43.520155 29307 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786332336319544-28292-0/minicluster-data/master-0/data/instance:
uuid: "71e49f3c3b794fb79e455a3f6656fb68"
format_stamp: "Formatted at 2026-08-10 03:31:43 on dist-test-slave-vpvm"
server_key: "fdcf53a4bb10f5fc58c3dd93e6efaacd"
server_key_iv: "48a051dd89c985f997a3f8597bbffc09"
server_key_version: "encryptionkey@0"
I20260810 03:31:43.520929 29307 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786332336319544-28292-0/minicluster-data/master-0/wal/instance:
uuid: "71e49f3c3b794fb79e455a3f6656fb68"
format_stamp: "Formatted at 2026-08-10 03:31:43 on dist-test-slave-vpvm"
server_key: "fdcf53a4bb10f5fc58c3dd93e6efaacd"
server_key_iv: "48a051dd89c985f997a3f8597bbffc09"
server_key_version: "encryptionkey@0"
I20260810 03:31:43.526340 29307 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260810 03:31:43.529922 29334 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:31:43.531294 29307 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 03:31:43.531462 29307 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786332336319544-28292-0/minicluster-data/master-0/data,/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786332336319544-28292-0/minicluster-data/master-0/wal
uuid: "71e49f3c3b794fb79e455a3f6656fb68"
format_stamp: "Formatted at 2026-08-10 03:31:43 on dist-test-slave-vpvm"
server_key: "fdcf53a4bb10f5fc58c3dd93e6efaacd"
server_key_iv: "48a051dd89c985f997a3f8597bbffc09"
server_key_version: "encryptionkey@0"
I20260810 03:31:43.531656 29307 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786332336319544-28292-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786332336319544-28292-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786332336319544-28292-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:31:43.545261 29307 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:31:43.549494 29307 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:31:43.549769 29307 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:31:43.569218 29307 rpc_server.cc:307] RPC server started. Bound to: 127.27.161.62:34437
I20260810 03:31:43.569226 29403 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.161.62:34437 every 8 connection(s)
I20260810 03:31:43.570734 29307 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786332336319544-28292-0/minicluster-data/master-0/data/info.pb
I20260810 03:31:43.574640 29404 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:31:43.578299 28292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iBH5/build/asan/bin/kudu as pid 29307
I20260810 03:31:43.578503 28292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786332336319544-28292-0/minicluster-data/master-0/wal/instance
I20260810 03:31:43.579048 28292 external_mini_cluster.cc:1428] Setting key d7e5798e913adfd672e9f7b9ccc580e7
I20260810 03:31:43.587929 29404 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 71e49f3c3b794fb79e455a3f6656fb68: Bootstrap starting.
I20260810 03:31:43.591414 29404 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 71e49f3c3b794fb79e455a3f6656fb68: Neither blocks nor log segments found. Creating new log.
I20260810 03:31:43.592530 29404 log.cc:824] T 00000000000000000000000000000000 P 71e49f3c3b794fb79e455a3f6656fb68: Log is configured to *not* fsync() on all Append() calls
Aug 10 03:31:43 dist-test-slave-vpvm krb5kdc[29289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332702, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.161.62@KRBTEST.COM
I20260810 03:31:43.595693 29404 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 71e49f3c3b794fb79e455a3f6656fb68: No bootstrap required, opened a new log
I20260810 03:31:43.603715 29411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:31:43.581077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42433 (local address 127.27.161.62:34437)
0810 03:31:43.582249 (+ 1172us) server_negotiation.cc:206] Beginning negotiation
0810 03:31:43.582265 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 03:31:43.582322 (+ 57us) server_negotiation.cc:407] Connection header received
0810 03:31:43.584179 (+ 1857us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:31:43.584205 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:31:43.584686 (+ 481us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:31:43.585130 (+ 444us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:31:43.586809 (+ 1679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:31:43.588292 (+ 1483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:31:43.589676 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:31:43.590448 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:31:43.590640 (+ 192us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:31:43.595100 (+ 4460us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:31:43.595109 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:31:43.595126 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:31:43.595164 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:31:43.598973 (+ 3809us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:31:43.599901 (+ 928us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:31:43.599910 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:31:43.599918 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:31:43.600130 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:31:43.600637 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:31:43.600641 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:31:43.600643 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:31:43.601258 (+ 615us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:31:43.601614 (+ 356us) server_negotiation.cc:1035] Waiting for connection context
0810 03:31:43.602194 (+ 580us) server_negotiation.cc:299] Negotiation successful
0810 03:31:43.602566 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":872,"thread_start_us":494,"threads_started":1}
I20260810 03:31:43.604346 29404 raft_consensus.cc:348] T 00000000000000000000000000000000 P 71e49f3c3b794fb79e455a3f6656fb68 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71e49f3c3b794fb79e455a3f6656fb68" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 34437 } }
I20260810 03:31:43.604764 29404 raft_consensus.cc:374] T 00000000000000000000000000000000 P 71e49f3c3b794fb79e455a3f6656fb68 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:31:43.604862 29404 raft_consensus.cc:729] T 00000000000000000000000000000000 P 71e49f3c3b794fb79e455a3f6656fb68 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71e49f3c3b794fb79e455a3f6656fb68, State: Initialized, Role: FOLLOWER
I20260810 03:31:43.605443 29404 consensus_queue.cc:260] T 00000000000000000000000000000000 P 71e49f3c3b794fb79e455a3f6656fb68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71e49f3c3b794fb79e455a3f6656fb68" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 34437 } }
I20260810 03:31:43.605698 29404 raft_consensus.cc:388] T 00000000000000000000000000000000 P 71e49f3c3b794fb79e455a3f6656fb68 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 03:31:43.605839 29404 raft_consensus.cc:482] T 00000000000000000000000000000000 P 71e49f3c3b794fb79e455a3f6656fb68 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 03:31:43.606042 29404 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 71e49f3c3b794fb79e455a3f6656fb68 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:31:43.608290 28292 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 03:31:43.609508 29404 raft_consensus.cc:504] T 00000000000000000000000000000000 P 71e49f3c3b794fb79e455a3f6656fb68 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71e49f3c3b794fb79e455a3f6656fb68" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 34437 } }
I20260810 03:31:43.610266 29404 leader_election.cc:302] T 00000000000000000000000000000000 P 71e49f3c3b794fb79e455a3f6656fb68 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 71e49f3c3b794fb79e455a3f6656fb68; no voters:
I20260810 03:31:43.610967 29404 leader_election.cc:288] T 00000000000000000000000000000000 P 71e49f3c3b794fb79e455a3f6656fb68 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 03:31:43.611101 29417 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 71e49f3c3b794fb79e455a3f6656fb68 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:31:43.612432 29417 raft_consensus.cc:686] T 00000000000000000000000000000000 P 71e49f3c3b794fb79e455a3f6656fb68 [term 1 LEADER]: Becoming Leader. State: Replica: 71e49f3c3b794fb79e455a3f6656fb68, State: Running, Role: LEADER
I20260810 03:31:43.613160 29417 consensus_queue.cc:237] T 00000000000000000000000000000000 P 71e49f3c3b794fb79e455a3f6656fb68 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71e49f3c3b794fb79e455a3f6656fb68" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 34437 } }
I20260810 03:31:43.613880 29404 sys_catalog.cc:564] T 00000000000000000000000000000000 P 71e49f3c3b794fb79e455a3f6656fb68 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:31:43.616526 29423 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71e49f3c3b794fb79e455a3f6656fb68 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 71e49f3c3b794fb79e455a3f6656fb68. Latest consensus state: current_term: 1 leader_uuid: "71e49f3c3b794fb79e455a3f6656fb68" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71e49f3c3b794fb79e455a3f6656fb68" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 34437 } } }
I20260810 03:31:43.617012 29423 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71e49f3c3b794fb79e455a3f6656fb68 [sys.catalog]: This master's current role is: LEADER
I20260810 03:31:43.617621 29422 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71e49f3c3b794fb79e455a3f6656fb68 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "71e49f3c3b794fb79e455a3f6656fb68" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71e49f3c3b794fb79e455a3f6656fb68" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 34437 } } }
I20260810 03:31:43.617945 29422 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71e49f3c3b794fb79e455a3f6656fb68 [sys.catalog]: This master's current role is: LEADER
I20260810 03:31:43.634939 29411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:31:43.614315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60859 (local address 127.27.161.62:34437)
0810 03:31:43.614530 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0810 03:31:43.614537 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:31:43.615081 (+ 544us) server_negotiation.cc:407] Connection header received
0810 03:31:43.615391 (+ 310us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:31:43.615396 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:31:43.615484 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:31:43.615598 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:31:43.617863 (+ 2265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:31:43.619228 (+ 1365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:31:43.621352 (+ 2124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:31:43.622137 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:31:43.622252 (+ 115us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:31:43.626339 (+ 4087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:31:43.626348 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:31:43.626354 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:31:43.626414 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:31:43.630405 (+ 3991us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:31:43.631726 (+ 1321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:31:43.631734 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:31:43.631738 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:31:43.631939 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:31:43.632606 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:31:43.632612 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:31:43.632615 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:31:43.633099 (+ 484us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:31:43.633380 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0810 03:31:43.634235 (+ 855us) server_negotiation.cc:299] Negotiation successful
0810 03:31:43.634501 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260810 03:31:43.640014 29428 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 03:31:43.651945 29428 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 03:31:43.663009 29428 catalog_manager.cc:1347] Generated new cluster ID: af67cce84bc746d69e7cb360d6e11b5b
I20260810 03:31:43.663149 29428 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 03:31:43.676438 29428 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 03:31:43.677935 29428 catalog_manager.cc:1504] Loading token signing keys...
I20260810 03:31:43.692659 29428 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 71e49f3c3b794fb79e455a3f6656fb68: Generated new TSK 0
I20260810 03:31:43.693935 29428 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 03:31:43.703945 28292 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iBH5/build/asan/bin/kudu pid 29307
I20260810 03:31:43.705184 29396 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:31:43.797731 28292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iBH5/build/asan/bin/kudu with pid 29307
W20260810 03:31:43.824412 29286 connection.cc:535] client connection to 127.27.161.62:34437 recv error: Network error: failed to read from TLS socket (remote: 127.27.161.62:34437): Connection reset by peer (error 104)
2026-08-10T03:31:43Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (2326 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786332336319544-28292-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 03:31:43 dist-test-slave-vpvm krb5kdc[29458](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 03:31:43 dist-test-slave-vpvm krb5kdc[29458](info): set up 2 sockets
Aug 10 03:31:43 dist-test-slave-vpvm krb5kdc[29458](info): commencing operation
krb5kdc: starting...
W20260810 03:31:45.054322 28292 mini_kdc.cc:121] Time spent starting KDC: real 1.211s user 0.000s sys 0.059s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786332336319544-28292-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786332336319544-28292-0/krb5kdc/test-user.keytab.
Aug 10 03:31:45 dist-test-slave-vpvm krb5kdc[29458](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332705, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T03:31:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T03:31:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.161.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.161.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.161.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.62.keytab.
Entry for principal kudu/127.27.161.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.62.keytab.
I20260810 03:31:45.413434 28292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iBH5/build/asan/bin/kudu
/tmp/dist-test-task24iBH5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786332336319544-28292-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786332336319544-28292-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786332336319544-28292-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786332336319544-28292-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.161.62:43245
--webserver_interface=127.27.161.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.62.keytab
--principal=kudu/127.27.161.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.161.20:43717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.161.62:43245
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786332336319544-28292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786332336319544-28292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786332336319544-28292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:31:45.788546 29474 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:31:45.788854 29474 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:31:45.788992 29474 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:31:45.796591 29474 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:31:45.796721 29474 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:31:45.796797 29474 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:31:45.796892 29474 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:31:45.796950 29474 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:31:45.807432 29474 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.161.20:43717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786332336319544-28292-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786332336319544-28292-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.161.62:43245
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.62.keytab
--principal=kudu/127.27.161.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.161.62:43245
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786332336319544-28292-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.161.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786332336319544-28292-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:31:45.808802 29474 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:31:45.810182 29474 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:31:45.822340 29484 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:31:45.822405 29485 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:31:45.822382 29488 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:31:45.823283 29474 server_base.cc:1034] running on GCE node
I20260810 03:31:45.824128 29474 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:31:45.825568 29474 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:31:45.826803 29474 hybrid_clock.cc:648] HybridClock initialized: now 1786332705826736 us; error 52 us; skew 500 ppm
I20260810 03:31:45.827162 29474 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:31:45 dist-test-slave-vpvm krb5kdc[29458](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332705, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.161.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:31:45.830413 29474 init.cc:377] Logged in from keytab as kudu/127.27.161.62@KRBTEST.COM (short username kudu)
I20260810 03:31:45.831948 29474 webserver.cc:460] Webserver started at http://127.27.161.62:41899/ using document root <none> and password file <none>
I20260810 03:31:45.832634 29474 fs_manager.cc:359] Metadata directory not provided
I20260810 03:31:45.832736 29474 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:31:45.833004 29474 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:31:45.836095 29474 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786332336319544-28292-0/minicluster-data/master-0/data/instance:
uuid: "2ceb890df374477d969659cb500ed58b"
format_stamp: "Formatted at 2026-08-10 03:31:45 on dist-test-slave-vpvm"
server_key: "01c9326717f82f6d3f7f127a5d03f76f"
server_key_iv: "f7389c1b87d7e47b2b2b056ad6bac5a3"
server_key_version: "encryptionkey@0"
I20260810 03:31:45.836839 29474 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786332336319544-28292-0/minicluster-data/master-0/wal/instance:
uuid: "2ceb890df374477d969659cb500ed58b"
format_stamp: "Formatted at 2026-08-10 03:31:45 on dist-test-slave-vpvm"
server_key: "01c9326717f82f6d3f7f127a5d03f76f"
server_key_iv: "f7389c1b87d7e47b2b2b056ad6bac5a3"
server_key_version: "encryptionkey@0"
I20260810 03:31:45.842136 29474 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260810 03:31:45.845651 29496 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:31:45.847035 29474 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 03:31:45.847190 29474 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786332336319544-28292-0/minicluster-data/master-0/data,/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786332336319544-28292-0/minicluster-data/master-0/wal
uuid: "2ceb890df374477d969659cb500ed58b"
format_stamp: "Formatted at 2026-08-10 03:31:45 on dist-test-slave-vpvm"
server_key: "01c9326717f82f6d3f7f127a5d03f76f"
server_key_iv: "f7389c1b87d7e47b2b2b056ad6bac5a3"
server_key_version: "encryptionkey@0"
I20260810 03:31:45.847362 29474 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786332336319544-28292-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786332336319544-28292-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786332336319544-28292-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:31:45.867111 29474 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:31:45.871488 29474 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:31:45.871749 29474 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:31:45.890041 29474 rpc_server.cc:307] RPC server started. Bound to: 127.27.161.62:43245
I20260810 03:31:45.890053 29576 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.161.62:43245 every 8 connection(s)
I20260810 03:31:45.891458 29474 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786332336319544-28292-0/minicluster-data/master-0/data/info.pb
I20260810 03:31:45.893343 28292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iBH5/build/asan/bin/kudu as pid 29474
I20260810 03:31:45.893527 28292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786332336319544-28292-0/minicluster-data/master-0/wal/instance
I20260810 03:31:45.894002 28292 external_mini_cluster.cc:1428] Setting key 2be3184d3dd20547155538507729dd45
I20260810 03:31:45.895831 29577 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:31:45.907670 29580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:31:45.895792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55559 (local address 127.27.161.62:43245)
0810 03:31:45.896602 (+ 810us) server_negotiation.cc:206] Beginning negotiation
0810 03:31:45.896614 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 03:31:45.896656 (+ 42us) server_negotiation.cc:407] Connection header received
0810 03:31:45.898329 (+ 1673us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:31:45.898354 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:31:45.898792 (+ 438us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:31:45.899146 (+ 354us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:31:45.899792 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:31:45.901150 (+ 1358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:31:45.902984 (+ 1834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:31:45.903691 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:31:45.903819 (+ 128us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:31:45.905175 (+ 1356us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:31:45.905187 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:31:45.905202 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:31:45.905823 (+ 621us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:31:45.905904 (+ 81us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:31:45.905935 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:31:45.906128 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0810 03:31:45.906402 (+ 274us) server_negotiation.cc:299] Negotiation successful
0810 03:31:45.906637 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":590,"thread_start_us":350,"threads_started":1}
I20260810 03:31:45.908977 29577 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ceb890df374477d969659cb500ed58b: Bootstrap starting.
I20260810 03:31:45.911777 28292 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 03:31:45.912384 29577 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2ceb890df374477d969659cb500ed58b: Neither blocks nor log segments found. Creating new log.
I20260810 03:31:45.913385 29577 log.cc:824] T 00000000000000000000000000000000 P 2ceb890df374477d969659cb500ed58b: Log is configured to *not* fsync() on all Append() calls
I20260810 03:31:45.916782 29577 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ceb890df374477d969659cb500ed58b: No bootstrap required, opened a new log
I20260810 03:31:45.922339 29580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:31:45.916242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34061 (local address 127.27.161.62:43245)
0810 03:31:45.916398 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0810 03:31:45.916404 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:31:45.916848 (+ 444us) server_negotiation.cc:407] Connection header received
0810 03:31:45.917043 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:31:45.917047 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:31:45.917109 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:31:45.917200 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:31:45.917703 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:31:45.918523 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:31:45.919854 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:31:45.920327 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:31:45.920437 (+ 110us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:31:45.921563 (+ 1126us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:31:45.921568 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:31:45.921571 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:31:45.921590 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:31:45.921635 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:31:45.921651 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:31:45.921762 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0810 03:31:45.921945 (+ 183us) server_negotiation.cc:299] Negotiation successful
0810 03:31:45.922040 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260810 03:31:45.924829 29577 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2ceb890df374477d969659cb500ed58b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ceb890df374477d969659cb500ed58b" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 43245 } }
I20260810 03:31:45.925271 29577 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2ceb890df374477d969659cb500ed58b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:31:45.925390 29577 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2ceb890df374477d969659cb500ed58b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ceb890df374477d969659cb500ed58b, State: Initialized, Role: FOLLOWER
I20260810 03:31:45.926088 29577 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2ceb890df374477d969659cb500ed58b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ceb890df374477d969659cb500ed58b" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 43245 } }
I20260810 03:31:45.926318 29577 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2ceb890df374477d969659cb500ed58b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 03:31:45.926424 29577 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2ceb890df374477d969659cb500ed58b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 03:31:45.926555 29577 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2ceb890df374477d969659cb500ed58b [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:31:45.928706 29577 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2ceb890df374477d969659cb500ed58b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ceb890df374477d969659cb500ed58b" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 43245 } }
I20260810 03:31:45.929162 29577 leader_election.cc:302] T 00000000000000000000000000000000 P 2ceb890df374477d969659cb500ed58b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2ceb890df374477d969659cb500ed58b; no voters:
I20260810 03:31:45.929731 29577 leader_election.cc:288] T 00000000000000000000000000000000 P 2ceb890df374477d969659cb500ed58b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 03:31:45.929858 29593 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2ceb890df374477d969659cb500ed58b [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:31:45.930543 29593 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2ceb890df374477d969659cb500ed58b [term 1 LEADER]: Becoming Leader. State: Replica: 2ceb890df374477d969659cb500ed58b, State: Running, Role: LEADER
I20260810 03:31:45.931058 29593 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2ceb890df374477d969659cb500ed58b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ceb890df374477d969659cb500ed58b" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 43245 } }
I20260810 03:31:45.931816 29577 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2ceb890df374477d969659cb500ed58b [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:31:45.933676 29595 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ceb890df374477d969659cb500ed58b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2ceb890df374477d969659cb500ed58b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ceb890df374477d969659cb500ed58b" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 43245 } } }
I20260810 03:31:45.934085 29595 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ceb890df374477d969659cb500ed58b [sys.catalog]: This master's current role is: LEADER
I20260810 03:31:45.934581 29596 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ceb890df374477d969659cb500ed58b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2ceb890df374477d969659cb500ed58b. Latest consensus state: current_term: 1 leader_uuid: "2ceb890df374477d969659cb500ed58b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ceb890df374477d969659cb500ed58b" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 43245 } } }
I20260810 03:31:45.934877 29596 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ceb890df374477d969659cb500ed58b [sys.catalog]: This master's current role is: LEADER
I20260810 03:31:45.934991 29603 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 03:31:45.942269 29603 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 03:31:45.953573 29603 catalog_manager.cc:1347] Generated new cluster ID: eb7c180c192a4e42a41f6def7f5c7bcb
I20260810 03:31:45.953713 29603 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 03:31:45.964112 29603 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 03:31:45.965397 29603 catalog_manager.cc:1504] Loading token signing keys...
I20260810 03:31:45.982726 29603 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2ceb890df374477d969659cb500ed58b: Generated new TSK 0
I20260810 03:31:45.984052 29603 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 03:31:45.990849 28292 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iBH5/build/asan/bin/kudu pid 29474
W20260810 03:31:45.992237 29570 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:55559
W20260810 03:31:45.993165 28292 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task24iBH5/build/asan/bin/kudu pid 29474: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260810 03:31:45.993330 28292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iBH5/build/asan/bin/kudu with pid 29474
W20260810 03:31:46.021649 29456 connection.cc:535] client connection to 127.27.161.62:43245 recv error: Network error: failed to read from TLS socket (remote: 127.27.161.62:43245): Connection reset by peer (error 104)
2026-08-10T03:31:46Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (2197 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786332336319544-28292-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 03:31:46 dist-test-slave-vpvm krb5kdc[29623](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 03:31:46 dist-test-slave-vpvm krb5kdc[29623](info): set up 2 sockets
Aug 10 03:31:46 dist-test-slave-vpvm krb5kdc[29623](info): commencing operation
krb5kdc: starting...
W20260810 03:31:47.261085 28292 mini_kdc.cc:121] Time spent starting KDC: real 1.220s user 0.002s sys 0.058s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786332336319544-28292-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786332336319544-28292-0/krb5kdc/test-user.keytab.
Aug 10 03:31:47 dist-test-slave-vpvm krb5kdc[29623](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332707, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T03:31:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T03:31:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.161.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.161.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.161.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.62.keytab.
Entry for principal kudu/127.27.161.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.62.keytab.
I20260810 03:31:47.622279 28292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iBH5/build/asan/bin/kudu
/tmp/dist-test-task24iBH5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786332336319544-28292-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786332336319544-28292-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786332336319544-28292-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786332336319544-28292-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.161.62:38671
--webserver_interface=127.27.161.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.62.keytab
--principal=kudu/127.27.161.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.161.20:38977
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.161.62:38671
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786332336319544-28292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786332336319544-28292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786332336319544-28292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:31:47.982491 29642 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:31:47.982825 29642 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:31:47.982946 29642 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:31:47.990500 29642 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:31:47.990629 29642 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:31:47.990696 29642 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:31:47.990783 29642 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:31:47.990840 29642 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:31:48.001283 29642 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.161.20:38977
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786332336319544-28292-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786332336319544-28292-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.161.62:38671
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.62.keytab
--principal=kudu/127.27.161.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.161.62:38671
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786332336319544-28292-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.161.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786332336319544-28292-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:31:48.002708 29642 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:31:48.004058 29642 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:31:48.015981 29649 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:31:48.016024 29650 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:31:48.016045 29654 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:31:48.016849 29642 server_base.cc:1034] running on GCE node
I20260810 03:31:48.017625 29642 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:31:48.019212 29642 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:31:48.020422 29642 hybrid_clock.cc:648] HybridClock initialized: now 1786332708020373 us; error 44 us; skew 500 ppm
I20260810 03:31:48.020769 29642 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:31:48 dist-test-slave-vpvm krb5kdc[29623](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332708, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.161.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:31:48.028234 29642 init.cc:377] Logged in from keytab as kudu/127.27.161.62@KRBTEST.COM (short username kudu)
I20260810 03:31:48.030002 29642 webserver.cc:460] Webserver started at http://127.27.161.62:34335/ using document root <none> and password file <none>
I20260810 03:31:48.030938 29642 fs_manager.cc:359] Metadata directory not provided
I20260810 03:31:48.031073 29642 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:31:48.031447 29642 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:31:48.034935 29642 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786332336319544-28292-0/minicluster-data/master-0/data/instance:
uuid: "9350cf1b52cd45439249838a8f87747a"
format_stamp: "Formatted at 2026-08-10 03:31:48 on dist-test-slave-vpvm"
server_key: "5c5864183274e34cedf3e66da3058423"
server_key_iv: "c37ec396a51e7e431428aa7a4498a187"
server_key_version: "encryptionkey@0"
I20260810 03:31:48.035769 29642 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786332336319544-28292-0/minicluster-data/master-0/wal/instance:
uuid: "9350cf1b52cd45439249838a8f87747a"
format_stamp: "Formatted at 2026-08-10 03:31:48 on dist-test-slave-vpvm"
server_key: "5c5864183274e34cedf3e66da3058423"
server_key_iv: "c37ec396a51e7e431428aa7a4498a187"
server_key_version: "encryptionkey@0"
I20260810 03:31:48.041482 29642 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260810 03:31:48.045295 29665 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:31:48.046746 29642 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 03:31:48.046916 29642 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786332336319544-28292-0/minicluster-data/master-0/data,/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786332336319544-28292-0/minicluster-data/master-0/wal
uuid: "9350cf1b52cd45439249838a8f87747a"
format_stamp: "Formatted at 2026-08-10 03:31:48 on dist-test-slave-vpvm"
server_key: "5c5864183274e34cedf3e66da3058423"
server_key_iv: "c37ec396a51e7e431428aa7a4498a187"
server_key_version: "encryptionkey@0"
I20260810 03:31:48.047127 29642 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786332336319544-28292-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786332336319544-28292-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786332336319544-28292-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:31:48.053275 29642 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:31:48.057636 29642 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:31:48.057940 29642 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:31:48.076995 29642 rpc_server.cc:307] RPC server started. Bound to: 127.27.161.62:38671
I20260810 03:31:48.077032 29734 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.161.62:38671 every 8 connection(s)
I20260810 03:31:48.078447 29642 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786332336319544-28292-0/minicluster-data/master-0/data/info.pb
I20260810 03:31:48.079519 28292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iBH5/build/asan/bin/kudu as pid 29642
I20260810 03:31:48.079707 28292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786332336319544-28292-0/minicluster-data/master-0/wal/instance
I20260810 03:31:48.080265 28292 external_mini_cluster.cc:1428] Setting key 76724e32185ec966c7d9cc47892fae09
I20260810 03:31:48.083845 29736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:31:48.088693 29740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:31:48.082387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56917 (local address 127.27.161.62:38671)
0810 03:31:48.083265 (+ 878us) server_negotiation.cc:206] Beginning negotiation
0810 03:31:48.083277 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 03:31:48.083319 (+ 42us) server_negotiation.cc:407] Connection header received
0810 03:31:48.085067 (+ 1748us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:31:48.085091 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:31:48.085499 (+ 408us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:31:48.085899 (+ 400us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:31:48.087022 (+ 1123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:31:48.087029 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:31:48.087042 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:31:48.087060 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:31:48.087148 (+ 88us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:31:48.087171 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:31:48.087406 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0810 03:31:48.087536 (+ 130us) server_negotiation.cc:299] Negotiation successful
0810 03:31:48.087744 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":599,"thread_start_us":350,"threads_started":1}
I20260810 03:31:48.092615 28292 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 03:31:48.096366 29736 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9350cf1b52cd45439249838a8f87747a: Bootstrap starting.
I20260810 03:31:48.099942 29740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:31:48.097206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40001 (local address 127.27.161.62:38671)
0810 03:31:48.097364 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0810 03:31:48.097368 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:31:48.097857 (+ 489us) server_negotiation.cc:407] Connection header received
0810 03:31:48.098064 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:31:48.098068 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:31:48.098126 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:31:48.098233 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:31:48.099253 (+ 1020us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:31:48.099258 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:31:48.099261 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:31:48.099279 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:31:48.099327 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:31:48.099344 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:31:48.099452 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0810 03:31:48.099591 (+ 139us) server_negotiation.cc:299] Negotiation successful
0810 03:31:48.099669 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260810 03:31:48.100590 29736 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9350cf1b52cd45439249838a8f87747a: Neither blocks nor log segments found. Creating new log.
I20260810 03:31:48.101927 29736 log.cc:824] T 00000000000000000000000000000000 P 9350cf1b52cd45439249838a8f87747a: Log is configured to *not* fsync() on all Append() calls
I20260810 03:31:48.104966 29736 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9350cf1b52cd45439249838a8f87747a: No bootstrap required, opened a new log
I20260810 03:31:48.113281 29736 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9350cf1b52cd45439249838a8f87747a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9350cf1b52cd45439249838a8f87747a" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 38671 } }
I20260810 03:31:48.113811 29736 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9350cf1b52cd45439249838a8f87747a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:31:48.113922 29736 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9350cf1b52cd45439249838a8f87747a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9350cf1b52cd45439249838a8f87747a, State: Initialized, Role: FOLLOWER
I20260810 03:31:48.114753 29736 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9350cf1b52cd45439249838a8f87747a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9350cf1b52cd45439249838a8f87747a" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 38671 } }
I20260810 03:31:48.115067 29736 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9350cf1b52cd45439249838a8f87747a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 03:31:48.115234 29736 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9350cf1b52cd45439249838a8f87747a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 03:31:48.115432 29736 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9350cf1b52cd45439249838a8f87747a [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:31:48.117913 29736 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9350cf1b52cd45439249838a8f87747a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9350cf1b52cd45439249838a8f87747a" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 38671 } }
I20260810 03:31:48.118562 29736 leader_election.cc:302] T 00000000000000000000000000000000 P 9350cf1b52cd45439249838a8f87747a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9350cf1b52cd45439249838a8f87747a; no voters:
I20260810 03:31:48.119231 29736 leader_election.cc:288] T 00000000000000000000000000000000 P 9350cf1b52cd45439249838a8f87747a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 03:31:48.119369 29749 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9350cf1b52cd45439249838a8f87747a [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:31:48.120078 29749 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9350cf1b52cd45439249838a8f87747a [term 1 LEADER]: Becoming Leader. State: Replica: 9350cf1b52cd45439249838a8f87747a, State: Running, Role: LEADER
I20260810 03:31:48.120765 29749 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9350cf1b52cd45439249838a8f87747a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9350cf1b52cd45439249838a8f87747a" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 38671 } }
I20260810 03:31:48.121733 29736 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9350cf1b52cd45439249838a8f87747a [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:31:48.125913 29751 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9350cf1b52cd45439249838a8f87747a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9350cf1b52cd45439249838a8f87747a. Latest consensus state: current_term: 1 leader_uuid: "9350cf1b52cd45439249838a8f87747a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9350cf1b52cd45439249838a8f87747a" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 38671 } } }
I20260810 03:31:48.126094 29750 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9350cf1b52cd45439249838a8f87747a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9350cf1b52cd45439249838a8f87747a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9350cf1b52cd45439249838a8f87747a" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 38671 } } }
I20260810 03:31:48.126441 29751 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9350cf1b52cd45439249838a8f87747a [sys.catalog]: This master's current role is: LEADER
I20260810 03:31:48.126492 29750 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9350cf1b52cd45439249838a8f87747a [sys.catalog]: This master's current role is: LEADER
I20260810 03:31:48.128094 29762 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 03:31:48.134300 29762 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 03:31:48.144268 29762 catalog_manager.cc:1347] Generated new cluster ID: f8202bba5c5e41a197f3234d2d4d9f5b
I20260810 03:31:48.144407 29762 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 03:31:48.169257 29762 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 03:31:48.169844 29762 catalog_manager.cc:1504] Loading token signing keys...
I20260810 03:31:48.181548 29762 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9350cf1b52cd45439249838a8f87747a: Generated new TSK 0
I20260810 03:31:48.182788 29762 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 03:31:48.190152 28292 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iBH5/build/asan/bin/kudu pid 29642
W20260810 03:31:48.191493 29729 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:56917
W20260810 03:31:48.192263 28292 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task24iBH5/build/asan/bin/kudu pid 29642: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260810 03:31:48.192402 28292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iBH5/build/asan/bin/kudu with pid 29642
2026-08-10T03:31:48Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (2198 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786332336319544-28292-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 03:31:48 dist-test-slave-vpvm krb5kdc[29783](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 03:31:48 dist-test-slave-vpvm krb5kdc[29783](info): set up 2 sockets
Aug 10 03:31:48 dist-test-slave-vpvm krb5kdc[29783](info): commencing operation
krb5kdc: starting...
W20260810 03:31:49.448829 28292 mini_kdc.cc:121] Time spent starting KDC: real 1.209s user 0.000s sys 0.058s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786332336319544-28292-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786332336319544-28292-0/krb5kdc/test-user.keytab.
Aug 10 03:31:49 dist-test-slave-vpvm krb5kdc[29783](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332709, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T03:31:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T03:31:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.161.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.161.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.161.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.62.keytab.
Entry for principal kudu/127.27.161.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.62.keytab.
I20260810 03:31:49.809752 28292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iBH5/build/asan/bin/kudu
/tmp/dist-test-task24iBH5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786332336319544-28292-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786332336319544-28292-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786332336319544-28292-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786332336319544-28292-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.161.62:40561
--webserver_interface=127.27.161.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.62.keytab
--principal=kudu/127.27.161.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.161.20:45419
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.161.62:40561
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786332336319544-28292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786332336319544-28292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786332336319544-28292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:31:50.166033 29803 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:31:50.166368 29803 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:31:50.166495 29803 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:31:50.174042 29803 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:31:50.174168 29803 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:31:50.174237 29803 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:31:50.174325 29803 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:31:50.174386 29803 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:31:50.184677 29803 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.161.20:45419
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786332336319544-28292-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786332336319544-28292-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.161.62:40561
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.62.keytab
--principal=kudu/127.27.161.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.161.62:40561
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786332336319544-28292-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.161.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786332336319544-28292-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:31:50.186079 29803 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:31:50.187497 29803 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:31:50.199743 29817 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:31:50.199731 29821 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:31:50.199671 29819 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:31:50.200912 29803 server_base.cc:1034] running on GCE node
I20260810 03:31:50.201840 29803 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:31:50.203563 29803 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:31:50.204825 29803 hybrid_clock.cc:648] HybridClock initialized: now 1786332710204762 us; error 46 us; skew 500 ppm
I20260810 03:31:50.205243 29803 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:31:50 dist-test-slave-vpvm krb5kdc[29783](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332710, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.161.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:31:50.209005 29803 init.cc:377] Logged in from keytab as kudu/127.27.161.62@KRBTEST.COM (short username kudu)
I20260810 03:31:50.211155 29803 webserver.cc:460] Webserver started at http://127.27.161.62:37389/ using document root <none> and password file <none>
I20260810 03:31:50.212141 29803 fs_manager.cc:359] Metadata directory not provided
I20260810 03:31:50.212275 29803 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:31:50.212607 29803 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:31:50.216025 29803 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786332336319544-28292-0/minicluster-data/master-0/data/instance:
uuid: "3a0c2d7c5f5e436f9f6bdc7aac2a1ff3"
format_stamp: "Formatted at 2026-08-10 03:31:50 on dist-test-slave-vpvm"
server_key: "0418fadabc5cfc62750c3f442db2fa5b"
server_key_iv: "d85c8411645030752550943348ca9253"
server_key_version: "encryptionkey@0"
I20260810 03:31:50.216883 29803 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786332336319544-28292-0/minicluster-data/master-0/wal/instance:
uuid: "3a0c2d7c5f5e436f9f6bdc7aac2a1ff3"
format_stamp: "Formatted at 2026-08-10 03:31:50 on dist-test-slave-vpvm"
server_key: "0418fadabc5cfc62750c3f442db2fa5b"
server_key_iv: "d85c8411645030752550943348ca9253"
server_key_version: "encryptionkey@0"
I20260810 03:31:50.222904 29803 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260810 03:31:50.226837 29831 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:31:50.228305 29803 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 03:31:50.228498 29803 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786332336319544-28292-0/minicluster-data/master-0/data,/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786332336319544-28292-0/minicluster-data/master-0/wal
uuid: "3a0c2d7c5f5e436f9f6bdc7aac2a1ff3"
format_stamp: "Formatted at 2026-08-10 03:31:50 on dist-test-slave-vpvm"
server_key: "0418fadabc5cfc62750c3f442db2fa5b"
server_key_iv: "d85c8411645030752550943348ca9253"
server_key_version: "encryptionkey@0"
I20260810 03:31:50.228716 29803 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786332336319544-28292-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786332336319544-28292-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786332336319544-28292-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:31:50.245944 29803 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:31:50.251219 29803 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:31:50.251585 29803 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:31:50.271922 29803 rpc_server.cc:307] RPC server started. Bound to: 127.27.161.62:40561
I20260810 03:31:50.271944 29909 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.161.62:40561 every 8 connection(s)
I20260810 03:31:50.273437 29803 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786332336319544-28292-0/minicluster-data/master-0/data/info.pb
I20260810 03:31:50.275949 28292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iBH5/build/asan/bin/kudu as pid 29803
I20260810 03:31:50.276142 28292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786332336319544-28292-0/minicluster-data/master-0/wal/instance
I20260810 03:31:50.276696 28292 external_mini_cluster.cc:1428] Setting key 2e32d0f09676d6485f26156e0798d071
I20260810 03:31:50.277969 29912 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 10 03:31:50 dist-test-slave-vpvm krb5kdc[29783](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332709, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.161.62@KRBTEST.COM
I20260810 03:31:50.290475 29912 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a0c2d7c5f5e436f9f6bdc7aac2a1ff3: Bootstrap starting.
I20260810 03:31:50.293991 29912 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3a0c2d7c5f5e436f9f6bdc7aac2a1ff3: Neither blocks nor log segments found. Creating new log.
I20260810 03:31:50.295082 29912 log.cc:824] T 00000000000000000000000000000000 P 3a0c2d7c5f5e436f9f6bdc7aac2a1ff3: Log is configured to *not* fsync() on all Append() calls
I20260810 03:31:50.298210 29912 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a0c2d7c5f5e436f9f6bdc7aac2a1ff3: No bootstrap required, opened a new log
I20260810 03:31:50.299175 29917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:31:50.278667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60119 (local address 127.27.161.62:40561)
0810 03:31:50.279484 (+ 817us) server_negotiation.cc:206] Beginning negotiation
0810 03:31:50.279496 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 03:31:50.279538 (+ 42us) server_negotiation.cc:407] Connection header received
0810 03:31:50.281168 (+ 1630us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:31:50.281192 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:31:50.281621 (+ 429us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:31:50.282030 (+ 409us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:31:50.283586 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:31:50.284947 (+ 1361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:31:50.286306 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:31:50.287013 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:31:50.287233 (+ 220us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:31:50.291495 (+ 4262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:31:50.291504 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:31:50.291520 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:31:50.291568 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:31:50.294701 (+ 3133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:31:50.295455 (+ 754us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:31:50.295462 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:31:50.295469 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:31:50.295701 (+ 232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:31:50.296125 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:31:50.296130 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:31:50.296133 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:31:50.296806 (+ 673us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:31:50.297113 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0810 03:31:50.297590 (+ 477us) server_negotiation.cc:299] Negotiation successful
0810 03:31:50.298080 (+ 490us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":594,"thread_start_us":371,"threads_started":1}
I20260810 03:31:50.303032 28292 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 03:31:50.306371 29912 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3a0c2d7c5f5e436f9f6bdc7aac2a1ff3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a0c2d7c5f5e436f9f6bdc7aac2a1ff3" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 40561 } }
I20260810 03:31:50.306792 29912 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3a0c2d7c5f5e436f9f6bdc7aac2a1ff3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:31:50.306883 29912 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3a0c2d7c5f5e436f9f6bdc7aac2a1ff3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a0c2d7c5f5e436f9f6bdc7aac2a1ff3, State: Initialized, Role: FOLLOWER
I20260810 03:31:50.307574 29912 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3a0c2d7c5f5e436f9f6bdc7aac2a1ff3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a0c2d7c5f5e436f9f6bdc7aac2a1ff3" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 40561 } }
I20260810 03:31:50.307822 29912 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3a0c2d7c5f5e436f9f6bdc7aac2a1ff3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 03:31:50.307962 29912 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3a0c2d7c5f5e436f9f6bdc7aac2a1ff3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 03:31:50.308152 29912 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3a0c2d7c5f5e436f9f6bdc7aac2a1ff3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:31:50.311051 29912 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3a0c2d7c5f5e436f9f6bdc7aac2a1ff3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a0c2d7c5f5e436f9f6bdc7aac2a1ff3" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 40561 } }
I20260810 03:31:50.311733 29912 leader_election.cc:302] T 00000000000000000000000000000000 P 3a0c2d7c5f5e436f9f6bdc7aac2a1ff3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3a0c2d7c5f5e436f9f6bdc7aac2a1ff3; no voters:
I20260810 03:31:50.312424 29912 leader_election.cc:288] T 00000000000000000000000000000000 P 3a0c2d7c5f5e436f9f6bdc7aac2a1ff3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 03:31:50.312546 29929 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3a0c2d7c5f5e436f9f6bdc7aac2a1ff3 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:31:50.313787 29929 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3a0c2d7c5f5e436f9f6bdc7aac2a1ff3 [term 1 LEADER]: Becoming Leader. State: Replica: 3a0c2d7c5f5e436f9f6bdc7aac2a1ff3, State: Running, Role: LEADER
I20260810 03:31:50.314503 29929 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3a0c2d7c5f5e436f9f6bdc7aac2a1ff3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a0c2d7c5f5e436f9f6bdc7aac2a1ff3" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 40561 } }
I20260810 03:31:50.314951 29912 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3a0c2d7c5f5e436f9f6bdc7aac2a1ff3 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:31:50.317353 29930 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a0c2d7c5f5e436f9f6bdc7aac2a1ff3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3a0c2d7c5f5e436f9f6bdc7aac2a1ff3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a0c2d7c5f5e436f9f6bdc7aac2a1ff3" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 40561 } } }
I20260810 03:31:50.317756 29930 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a0c2d7c5f5e436f9f6bdc7aac2a1ff3 [sys.catalog]: This master's current role is: LEADER
I20260810 03:31:50.319792 29931 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a0c2d7c5f5e436f9f6bdc7aac2a1ff3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3a0c2d7c5f5e436f9f6bdc7aac2a1ff3. Latest consensus state: current_term: 1 leader_uuid: "3a0c2d7c5f5e436f9f6bdc7aac2a1ff3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a0c2d7c5f5e436f9f6bdc7aac2a1ff3" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 40561 } } }
I20260810 03:31:50.320163 29931 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a0c2d7c5f5e436f9f6bdc7aac2a1ff3 [sys.catalog]: This master's current role is: LEADER
I20260810 03:31:50.322667 29935 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 03:31:50.325091 29917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:31:50.307443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47225 (local address 127.27.161.62:40561)
0810 03:31:50.307640 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0810 03:31:50.307647 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:31:50.307980 (+ 333us) server_negotiation.cc:407] Connection header received
0810 03:31:50.308161 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:31:50.308167 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:31:50.308255 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:31:50.308371 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:31:50.309828 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:31:50.310930 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:31:50.312328 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:31:50.313105 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:31:50.313244 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:31:50.316715 (+ 3471us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:31:50.316723 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:31:50.316729 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:31:50.316783 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:31:50.321024 (+ 4241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:31:50.322169 (+ 1145us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:31:50.322175 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:31:50.322178 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:31:50.322383 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:31:50.323007 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:31:50.323012 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:31:50.323015 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:31:50.323432 (+ 417us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:31:50.323755 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0810 03:31:50.324215 (+ 460us) server_negotiation.cc:299] Negotiation successful
0810 03:31:50.324460 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260810 03:31:50.328984 29935 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 03:31:50.338788 29935 catalog_manager.cc:1347] Generated new cluster ID: a4f064e66e144d52ad23839953c8e621
I20260810 03:31:50.338903 29935 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 03:31:50.348914 29935 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 03:31:50.350838 29935 catalog_manager.cc:1504] Loading token signing keys...
I20260810 03:31:50.364709 29935 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3a0c2d7c5f5e436f9f6bdc7aac2a1ff3: Generated new TSK 0
I20260810 03:31:50.365648 29935 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 03:31:50.371831 28292 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iBH5/build/asan/bin/kudu pid 29803
I20260810 03:31:50.373088 29902 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:31:50.462529 28292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iBH5/build/asan/bin/kudu with pid 29803
2026-08-10T03:31:50Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (2264 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786332336319544-28292-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 03:31:50 dist-test-slave-vpvm krb5kdc[29959](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 03:31:50 dist-test-slave-vpvm krb5kdc[29959](info): set up 2 sockets
Aug 10 03:31:50 dist-test-slave-vpvm krb5kdc[29959](info): commencing operation
krb5kdc: starting...
W20260810 03:31:51.719245 28292 mini_kdc.cc:121] Time spent starting KDC: real 1.215s user 0.000s sys 0.059s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786332336319544-28292-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786332336319544-28292-0/krb5kdc/test-user.keytab.
Aug 10 03:31:51 dist-test-slave-vpvm krb5kdc[29959](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332711, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T03:31:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T03:31:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.161.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.161.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.161.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.62.keytab.
Entry for principal kudu/127.27.161.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.62.keytab.
I20260810 03:31:52.087352 28292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iBH5/build/asan/bin/kudu
/tmp/dist-test-task24iBH5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786332336319544-28292-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786332336319544-28292-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786332336319544-28292-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786332336319544-28292-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.161.62:46191
--webserver_interface=127.27.161.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.62.keytab
--principal=kudu/127.27.161.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.161.20:40493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.161.62:46191
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786332336319544-28292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786332336319544-28292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786332336319544-28292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:31:52.446285 29981 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:31:52.446652 29981 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:31:52.446820 29981 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:31:52.454811 29981 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:31:52.454969 29981 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:31:52.455051 29981 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:31:52.455135 29981 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:31:52.455197 29981 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:31:52.465662 29981 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.161.20:40493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786332336319544-28292-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786332336319544-28292-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.161.62:46191
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.62.keytab
--principal=kudu/127.27.161.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.161.62:46191
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786332336319544-28292-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.161.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786332336319544-28292-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:31:52.467160 29981 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:31:52.468560 29981 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:31:52.481060 29992 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:31:52.480970 29988 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:31:52.481149 29989 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:31:52.482488 29981 server_base.cc:1034] running on GCE node
I20260810 03:31:52.483574 29981 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:31:52.485231 29981 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:31:52.486479 29981 hybrid_clock.cc:648] HybridClock initialized: now 1786332712486416 us; error 57 us; skew 500 ppm
I20260810 03:31:52.486901 29981 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:31:52 dist-test-slave-vpvm krb5kdc[29959](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332712, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.161.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:31:52.490435 29981 init.cc:377] Logged in from keytab as kudu/127.27.161.62@KRBTEST.COM (short username kudu)
I20260810 03:31:52.492252 29981 webserver.cc:460] Webserver started at http://127.27.161.62:41057/ using document root <none> and password file <none>
I20260810 03:31:52.493010 29981 fs_manager.cc:359] Metadata directory not provided
I20260810 03:31:52.493124 29981 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:31:52.493433 29981 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:31:52.496781 29981 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786332336319544-28292-0/minicluster-data/master-0/data/instance:
uuid: "09424f9ffcec4ad3ac84da791ae938bd"
format_stamp: "Formatted at 2026-08-10 03:31:52 on dist-test-slave-vpvm"
server_key: "2bbf168904952a6225696da65502fac6"
server_key_iv: "d461540e33d503405bf886aa6fe75ca1"
server_key_version: "encryptionkey@0"
I20260810 03:31:52.497627 29981 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786332336319544-28292-0/minicluster-data/master-0/wal/instance:
uuid: "09424f9ffcec4ad3ac84da791ae938bd"
format_stamp: "Formatted at 2026-08-10 03:31:52 on dist-test-slave-vpvm"
server_key: "2bbf168904952a6225696da65502fac6"
server_key_iv: "d461540e33d503405bf886aa6fe75ca1"
server_key_version: "encryptionkey@0"
I20260810 03:31:52.503862 29981 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.003s
I20260810 03:31:52.508011 30002 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:31:52.509552 29981 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 03:31:52.509747 29981 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786332336319544-28292-0/minicluster-data/master-0/data,/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786332336319544-28292-0/minicluster-data/master-0/wal
uuid: "09424f9ffcec4ad3ac84da791ae938bd"
format_stamp: "Formatted at 2026-08-10 03:31:52 on dist-test-slave-vpvm"
server_key: "2bbf168904952a6225696da65502fac6"
server_key_iv: "d461540e33d503405bf886aa6fe75ca1"
server_key_version: "encryptionkey@0"
I20260810 03:31:52.509984 29981 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786332336319544-28292-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786332336319544-28292-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786332336319544-28292-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:31:52.522941 29981 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:31:52.527679 29981 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:31:52.527949 29981 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:31:52.546896 29981 rpc_server.cc:307] RPC server started. Bound to: 127.27.161.62:46191
I20260810 03:31:52.546921 30085 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.161.62:46191 every 8 connection(s)
I20260810 03:31:52.548300 29981 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786332336319544-28292-0/minicluster-data/master-0/data/info.pb
I20260810 03:31:52.552230 30086 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:31:52.553409 28292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iBH5/build/asan/bin/kudu as pid 29981
I20260810 03:31:52.553609 28292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786332336319544-28292-0/minicluster-data/master-0/wal/instance
I20260810 03:31:52.554214 28292 external_mini_cluster.cc:1428] Setting key 01953ca32ebf00480f43478c7f28d0ec
I20260810 03:31:52.567715 30086 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 09424f9ffcec4ad3ac84da791ae938bd: Bootstrap starting.
I20260810 03:31:52.569825 30090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:31:52.556079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44287 (local address 127.27.161.62:46191)
0810 03:31:52.556953 (+ 874us) server_negotiation.cc:206] Beginning negotiation
0810 03:31:52.556974 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0810 03:31:52.557028 (+ 54us) server_negotiation.cc:407] Connection header received
0810 03:31:52.559188 (+ 2160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:31:52.559226 (+ 38us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:31:52.559852 (+ 626us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:31:52.560353 (+ 501us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:31:52.560895 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:31:52.562820 (+ 1925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:31:52.564364 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:31:52.565457 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:31:52.565642 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:31:52.566847 (+ 1205us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:31:52.566858 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:31:52.566878 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:31:52.567617 (+ 739us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:31:52.567728 (+ 111us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:31:52.567762 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:31:52.567879 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0810 03:31:52.568139 (+ 260us) server_negotiation.cc:299] Negotiation successful
0810 03:31:52.568456 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":600,"thread_start_us":359,"threads_started":1}
I20260810 03:31:52.571978 30086 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 09424f9ffcec4ad3ac84da791ae938bd: Neither blocks nor log segments found. Creating new log.
I20260810 03:31:52.573194 30086 log.cc:824] T 00000000000000000000000000000000 P 09424f9ffcec4ad3ac84da791ae938bd: Log is configured to *not* fsync() on all Append() calls
I20260810 03:31:52.573935 28292 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 03:31:52.576058 30086 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 09424f9ffcec4ad3ac84da791ae938bd: No bootstrap required, opened a new log
I20260810 03:31:52.585790 30090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:31:52.578304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33087 (local address 127.27.161.62:46191)
0810 03:31:52.578509 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0810 03:31:52.578520 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 03:31:52.578879 (+ 359us) server_negotiation.cc:407] Connection header received
0810 03:31:52.579074 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:31:52.579080 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:31:52.579147 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:31:52.579247 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:31:52.579895 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:31:52.580687 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:31:52.582740 (+ 2053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:31:52.583445 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:31:52.583631 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:31:52.584950 (+ 1319us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:31:52.584957 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:31:52.584961 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:31:52.584983 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:31:52.585045 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:31:52.585064 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:31:52.585149 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0810 03:31:52.585344 (+ 195us) server_negotiation.cc:299] Negotiation successful
0810 03:31:52.585448 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260810 03:31:52.586139 30086 raft_consensus.cc:348] T 00000000000000000000000000000000 P 09424f9ffcec4ad3ac84da791ae938bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09424f9ffcec4ad3ac84da791ae938bd" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 46191 } }
I20260810 03:31:52.586644 30086 raft_consensus.cc:374] T 00000000000000000000000000000000 P 09424f9ffcec4ad3ac84da791ae938bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:31:52.586771 30086 raft_consensus.cc:729] T 00000000000000000000000000000000 P 09424f9ffcec4ad3ac84da791ae938bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09424f9ffcec4ad3ac84da791ae938bd, State: Initialized, Role: FOLLOWER
I20260810 03:31:52.587405 30086 consensus_queue.cc:260] T 00000000000000000000000000000000 P 09424f9ffcec4ad3ac84da791ae938bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09424f9ffcec4ad3ac84da791ae938bd" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 46191 } }
I20260810 03:31:52.587644 30086 raft_consensus.cc:388] T 00000000000000000000000000000000 P 09424f9ffcec4ad3ac84da791ae938bd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 03:31:52.587781 30086 raft_consensus.cc:482] T 00000000000000000000000000000000 P 09424f9ffcec4ad3ac84da791ae938bd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 03:31:52.587934 30086 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 09424f9ffcec4ad3ac84da791ae938bd [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:31:52.590260 30086 raft_consensus.cc:504] T 00000000000000000000000000000000 P 09424f9ffcec4ad3ac84da791ae938bd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09424f9ffcec4ad3ac84da791ae938bd" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 46191 } }
I20260810 03:31:52.590809 30086 leader_election.cc:302] T 00000000000000000000000000000000 P 09424f9ffcec4ad3ac84da791ae938bd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 09424f9ffcec4ad3ac84da791ae938bd; no voters:
I20260810 03:31:52.591477 30086 leader_election.cc:288] T 00000000000000000000000000000000 P 09424f9ffcec4ad3ac84da791ae938bd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 03:31:52.591614 30099 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 09424f9ffcec4ad3ac84da791ae938bd [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:31:52.592309 30099 raft_consensus.cc:686] T 00000000000000000000000000000000 P 09424f9ffcec4ad3ac84da791ae938bd [term 1 LEADER]: Becoming Leader. State: Replica: 09424f9ffcec4ad3ac84da791ae938bd, State: Running, Role: LEADER
I20260810 03:31:52.592921 30099 consensus_queue.cc:237] T 00000000000000000000000000000000 P 09424f9ffcec4ad3ac84da791ae938bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09424f9ffcec4ad3ac84da791ae938bd" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 46191 } }
I20260810 03:31:52.594020 30086 sys_catalog.cc:564] T 00000000000000000000000000000000 P 09424f9ffcec4ad3ac84da791ae938bd [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:31:52.595813 30103 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09424f9ffcec4ad3ac84da791ae938bd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 09424f9ffcec4ad3ac84da791ae938bd. Latest consensus state: current_term: 1 leader_uuid: "09424f9ffcec4ad3ac84da791ae938bd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09424f9ffcec4ad3ac84da791ae938bd" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 46191 } } }
I20260810 03:31:52.596244 30103 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09424f9ffcec4ad3ac84da791ae938bd [sys.catalog]: This master's current role is: LEADER
I20260810 03:31:52.596997 30102 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09424f9ffcec4ad3ac84da791ae938bd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "09424f9ffcec4ad3ac84da791ae938bd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09424f9ffcec4ad3ac84da791ae938bd" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 46191 } } }
I20260810 03:31:52.597347 30102 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09424f9ffcec4ad3ac84da791ae938bd [sys.catalog]: This master's current role is: LEADER
I20260810 03:31:52.597216 30107 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 03:31:52.602705 30107 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 03:31:52.613188 30107 catalog_manager.cc:1347] Generated new cluster ID: da3424d110724b848f6315d48a6ce1b5
I20260810 03:31:52.613379 30107 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 03:31:52.629299 30107 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 03:31:52.630816 30107 catalog_manager.cc:1504] Loading token signing keys...
I20260810 03:31:52.648116 30107 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 09424f9ffcec4ad3ac84da791ae938bd: Generated new TSK 0
I20260810 03:31:52.649049 30107 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 03:31:52.666585 28292 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iBH5/build/asan/bin/kudu pid 29981
W20260810 03:31:52.667665 30076 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:44287
W20260810 03:31:52.668354 28292 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task24iBH5/build/asan/bin/kudu pid 29981: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260810 03:31:52.668485 28292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iBH5/build/asan/bin/kudu with pid 29981
2026-08-10T03:31:52Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (2208 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786332336319544-28292-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 03:31:52 dist-test-slave-vpvm krb5kdc[30129](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 03:31:52 dist-test-slave-vpvm krb5kdc[30129](info): set up 2 sockets
Aug 10 03:31:52 dist-test-slave-vpvm krb5kdc[30129](info): commencing operation
krb5kdc: starting...
W20260810 03:31:53.931131 28292 mini_kdc.cc:121] Time spent starting KDC: real 1.219s user 0.003s sys 0.056s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786332336319544-28292-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786332336319544-28292-0/krb5kdc/test-user.keytab.
Aug 10 03:31:54 dist-test-slave-vpvm krb5kdc[30129](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332714, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T03:31:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T03:31:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.161.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.161.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.161.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.62.keytab.
Entry for principal kudu/127.27.161.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.62.keytab.
I20260810 03:31:54.297886 28292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iBH5/build/asan/bin/kudu
/tmp/dist-test-task24iBH5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786332336319544-28292-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786332336319544-28292-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786332336319544-28292-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786332336319544-28292-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.161.62:33595
--webserver_interface=127.27.161.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.62.keytab
--principal=kudu/127.27.161.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.161.20:43537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.161.62:33595
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786332336319544-28292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786332336319544-28292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786332336319544-28292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:31:54.662267 30159 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:31:54.662571 30159 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:31:54.662694 30159 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:31:54.670056 30159 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:31:54.670185 30159 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:31:54.670255 30159 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:31:54.670341 30159 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:31:54.670398 30159 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:31:54.680737 30159 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.161.20:43537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786332336319544-28292-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786332336319544-28292-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.161.62:33595
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.62.keytab
--principal=kudu/127.27.161.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.161.62:33595
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786332336319544-28292-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.161.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786332336319544-28292-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:31:54.682184 30159 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:31:54.683580 30159 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:31:54.695027 30165 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:31:54.695096 30166 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:31:54.695164 30168 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:31:54.696031 30159 server_base.cc:1034] running on GCE node
I20260810 03:31:54.696803 30159 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:31:54.698278 30159 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:31:54.699482 30159 hybrid_clock.cc:648] HybridClock initialized: now 1786332714699442 us; error 32 us; skew 500 ppm
I20260810 03:31:54.699829 30159 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:31:54 dist-test-slave-vpvm krb5kdc[30129](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332714, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.161.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:31:54.705626 30159 init.cc:377] Logged in from keytab as kudu/127.27.161.62@KRBTEST.COM (short username kudu)
I20260810 03:31:54.707144 30159 webserver.cc:460] Webserver started at http://127.27.161.62:33577/ using document root <none> and password file <none>
I20260810 03:31:54.707832 30159 fs_manager.cc:359] Metadata directory not provided
I20260810 03:31:54.707942 30159 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:31:54.708256 30159 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:31:54.711301 30159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786332336319544-28292-0/minicluster-data/master-0/data/instance:
uuid: "b8a9af5b58c647e6984a49ad5cdd106d"
format_stamp: "Formatted at 2026-08-10 03:31:54 on dist-test-slave-vpvm"
server_key: "4c378614e1cd55ccfda02bf1defac84e"
server_key_iv: "77c66b0b3d1b60ca18ad6d5f7be6aeaa"
server_key_version: "encryptionkey@0"
I20260810 03:31:54.712087 30159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786332336319544-28292-0/minicluster-data/master-0/wal/instance:
uuid: "b8a9af5b58c647e6984a49ad5cdd106d"
format_stamp: "Formatted at 2026-08-10 03:31:54 on dist-test-slave-vpvm"
server_key: "4c378614e1cd55ccfda02bf1defac84e"
server_key_iv: "77c66b0b3d1b60ca18ad6d5f7be6aeaa"
server_key_version: "encryptionkey@0"
I20260810 03:31:54.717264 30159 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260810 03:31:54.720731 30177 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:31:54.722100 30159 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 03:31:54.722275 30159 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786332336319544-28292-0/minicluster-data/master-0/data,/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786332336319544-28292-0/minicluster-data/master-0/wal
uuid: "b8a9af5b58c647e6984a49ad5cdd106d"
format_stamp: "Formatted at 2026-08-10 03:31:54 on dist-test-slave-vpvm"
server_key: "4c378614e1cd55ccfda02bf1defac84e"
server_key_iv: "77c66b0b3d1b60ca18ad6d5f7be6aeaa"
server_key_version: "encryptionkey@0"
I20260810 03:31:54.722462 30159 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786332336319544-28292-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786332336319544-28292-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786332336319544-28292-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:31:54.728055 30159 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:31:54.732259 30159 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:31:54.732540 30159 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:31:54.748905 30159 rpc_server.cc:307] RPC server started. Bound to: 127.27.161.62:33595
I20260810 03:31:54.748939 30259 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.161.62:33595 every 8 connection(s)
I20260810 03:31:54.750346 30159 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786332336319544-28292-0/minicluster-data/master-0/data/info.pb
I20260810 03:31:54.754307 30260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:31:54.754705 28292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iBH5/build/asan/bin/kudu as pid 30159
I20260810 03:31:54.754851 28292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786332336319544-28292-0/minicluster-data/master-0/wal/instance
I20260810 03:31:54.755285 28292 external_mini_cluster.cc:1428] Setting key 661dac3ecbe77fe6d78a01dbf4d0e264
I20260810 03:31:54.764492 30265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:31:54.757023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40557 (local address 127.27.161.62:33595)
0810 03:31:54.757884 (+ 861us) server_negotiation.cc:206] Beginning negotiation
0810 03:31:54.757898 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 03:31:54.757971 (+ 73us) server_negotiation.cc:407] Connection header received
0810 03:31:54.760697 (+ 2726us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:31:54.760727 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:31:54.761191 (+ 464us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:31:54.761657 (+ 466us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:31:54.762596 (+ 939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:31:54.762605 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:31:54.762621 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:31:54.762637 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:31:54.762707 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:31:54.762732 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:31:54.762938 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0810 03:31:54.763070 (+ 132us) server_negotiation.cc:299] Negotiation successful
0810 03:31:54.763282 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":574,"thread_start_us":337,"threads_started":1}
I20260810 03:31:54.768409 28292 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 03:31:54.768879 30260 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b8a9af5b58c647e6984a49ad5cdd106d: Bootstrap starting.
I20260810 03:31:54.772785 30260 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b8a9af5b58c647e6984a49ad5cdd106d: Neither blocks nor log segments found. Creating new log.
I20260810 03:31:54.774046 30260 log.cc:824] T 00000000000000000000000000000000 P b8a9af5b58c647e6984a49ad5cdd106d: Log is configured to *not* fsync() on all Append() calls
I20260810 03:31:54.775815 30265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:31:54.773510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43327 (local address 127.27.161.62:33595)
0810 03:31:54.773684 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0810 03:31:54.773689 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:31:54.774012 (+ 323us) server_negotiation.cc:407] Connection header received
0810 03:31:54.774184 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:31:54.774190 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:31:54.774252 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:31:54.774357 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:31:54.775208 (+ 851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:31:54.775212 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:31:54.775215 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:31:54.775230 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:31:54.775268 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:31:54.775285 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:31:54.775388 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0810 03:31:54.775496 (+ 108us) server_negotiation.cc:299] Negotiation successful
0810 03:31:54.775554 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260810 03:31:54.777275 30260 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b8a9af5b58c647e6984a49ad5cdd106d: No bootstrap required, opened a new log
I20260810 03:31:54.785130 30260 raft_consensus.cc:348] T 00000000000000000000000000000000 P b8a9af5b58c647e6984a49ad5cdd106d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8a9af5b58c647e6984a49ad5cdd106d" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 33595 } }
I20260810 03:31:54.785487 30260 raft_consensus.cc:374] T 00000000000000000000000000000000 P b8a9af5b58c647e6984a49ad5cdd106d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:31:54.785580 30260 raft_consensus.cc:729] T 00000000000000000000000000000000 P b8a9af5b58c647e6984a49ad5cdd106d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8a9af5b58c647e6984a49ad5cdd106d, State: Initialized, Role: FOLLOWER
I20260810 03:31:54.786147 30260 consensus_queue.cc:260] T 00000000000000000000000000000000 P b8a9af5b58c647e6984a49ad5cdd106d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8a9af5b58c647e6984a49ad5cdd106d" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 33595 } }
I20260810 03:31:54.786365 30260 raft_consensus.cc:388] T 00000000000000000000000000000000 P b8a9af5b58c647e6984a49ad5cdd106d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 03:31:54.786461 30260 raft_consensus.cc:482] T 00000000000000000000000000000000 P b8a9af5b58c647e6984a49ad5cdd106d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 03:31:54.786583 30260 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b8a9af5b58c647e6984a49ad5cdd106d [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:31:54.789008 30260 raft_consensus.cc:504] T 00000000000000000000000000000000 P b8a9af5b58c647e6984a49ad5cdd106d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8a9af5b58c647e6984a49ad5cdd106d" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 33595 } }
I20260810 03:31:54.789466 30260 leader_election.cc:302] T 00000000000000000000000000000000 P b8a9af5b58c647e6984a49ad5cdd106d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b8a9af5b58c647e6984a49ad5cdd106d; no voters:
I20260810 03:31:54.790035 30260 leader_election.cc:288] T 00000000000000000000000000000000 P b8a9af5b58c647e6984a49ad5cdd106d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 03:31:54.790143 30276 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b8a9af5b58c647e6984a49ad5cdd106d [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:31:54.790763 30276 raft_consensus.cc:686] T 00000000000000000000000000000000 P b8a9af5b58c647e6984a49ad5cdd106d [term 1 LEADER]: Becoming Leader. State: Replica: b8a9af5b58c647e6984a49ad5cdd106d, State: Running, Role: LEADER
I20260810 03:31:54.791309 30276 consensus_queue.cc:237] T 00000000000000000000000000000000 P b8a9af5b58c647e6984a49ad5cdd106d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8a9af5b58c647e6984a49ad5cdd106d" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 33595 } }
I20260810 03:31:54.792035 30260 sys_catalog.cc:564] T 00000000000000000000000000000000 P b8a9af5b58c647e6984a49ad5cdd106d [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:31:54.794976 30278 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8a9af5b58c647e6984a49ad5cdd106d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b8a9af5b58c647e6984a49ad5cdd106d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8a9af5b58c647e6984a49ad5cdd106d" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 33595 } } }
I20260810 03:31:54.795395 30278 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8a9af5b58c647e6984a49ad5cdd106d [sys.catalog]: This master's current role is: LEADER
I20260810 03:31:54.796146 30288 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 03:31:54.796066 30279 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8a9af5b58c647e6984a49ad5cdd106d [sys.catalog]: SysCatalogTable state changed. Reason: New leader b8a9af5b58c647e6984a49ad5cdd106d. Latest consensus state: current_term: 1 leader_uuid: "b8a9af5b58c647e6984a49ad5cdd106d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8a9af5b58c647e6984a49ad5cdd106d" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 33595 } } }
I20260810 03:31:54.796380 30279 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8a9af5b58c647e6984a49ad5cdd106d [sys.catalog]: This master's current role is: LEADER
I20260810 03:31:54.801167 30288 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 03:31:54.809502 30288 catalog_manager.cc:1347] Generated new cluster ID: 3d8bc49a7d3546c99c7e1ca8cbceee19
I20260810 03:31:54.809630 30288 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 03:31:54.823913 30288 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 03:31:54.824434 30288 catalog_manager.cc:1504] Loading token signing keys...
I20260810 03:31:54.834232 30288 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b8a9af5b58c647e6984a49ad5cdd106d: Generated new TSK 0
I20260810 03:31:54.835315 30288 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 03:31:54.851491 28292 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iBH5/build/asan/bin/kudu pid 30159
W20260810 03:31:54.852627 30252 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:40557
W20260810 03:31:54.853292 28292 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task24iBH5/build/asan/bin/kudu pid 30159: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260810 03:31:54.853448 28292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iBH5/build/asan/bin/kudu with pid 30159
2026-08-10T03:31:54Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (2183 ms)
[----------] 18 tests from AuthTokenIssuingTest (40186 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786332336319544-28292-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 03:31:54 dist-test-slave-vpvm krb5kdc[30312](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 03:31:54 dist-test-slave-vpvm krb5kdc[30312](info): set up 2 sockets
Aug 10 03:31:54 dist-test-slave-vpvm krb5kdc[30312](info): commencing operation
krb5kdc: starting...
W20260810 03:31:56.119498 28292 mini_kdc.cc:121] Time spent starting KDC: real 1.222s user 0.000s sys 0.060s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786332336319544-28292-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786332336319544-28292-0/krb5kdc/test-user.keytab.
Aug 10 03:31:56 dist-test-slave-vpvm krb5kdc[30312](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332716, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T03:31:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T03:31:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.161.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.161.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.161.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.62.keytab.
Entry for principal kudu/127.27.161.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.62.keytab.
I20260810 03:31:56.491438 28292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iBH5/build/asan/bin/kudu
/tmp/dist-test-task24iBH5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786332336319544-28292-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786332336319544-28292-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786332336319544-28292-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786332336319544-28292-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.161.62:43211
--webserver_interface=127.27.161.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.62.keytab
--principal=kudu/127.27.161.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.161.20:37217
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.161.62:43211,127.27.161.61:45273,127.27.161.60:40293
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786332336319544-28292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786332336319544-28292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786332336319544-28292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:31:56.851293 30330 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260810 03:31:56.851657 30330 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:31:56.851770 30330 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:31:56.851889 30330 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:31:56.859306 30330 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:31:56.859416 30330 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:31:56.859485 30330 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:31:56.859558 30330 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:31:56.859647 30330 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:31:56.870040 30330 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.161.20:37217
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786332336319544-28292-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786332336319544-28292-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.161.62:43211,127.27.161.61:45273,127.27.161.60:40293
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.62.keytab
--principal=kudu/127.27.161.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.161.62:43211
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786332336319544-28292-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.161.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786332336319544-28292-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:31:56.871462 30330 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:31:56.872830 30330 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:31:56.884696 30335 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:31:56.884822 30339 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:31:56.884812 30336 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:31:56.885783 30330 server_base.cc:1034] running on GCE node
I20260810 03:31:56.886662 30330 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:31:56.888207 30330 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:31:56.889460 30330 hybrid_clock.cc:648] HybridClock initialized: now 1786332716889377 us; error 46 us; skew 500 ppm
I20260810 03:31:56.889827 30330 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:31:56 dist-test-slave-vpvm krb5kdc[30312](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332716, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.161.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:31:56.893311 30330 init.cc:377] Logged in from keytab as kudu/127.27.161.62@KRBTEST.COM (short username kudu)
I20260810 03:31:56.895026 30330 webserver.cc:460] Webserver started at http://127.27.161.62:42829/ using document root <none> and password file <none>
I20260810 03:31:56.895766 30330 fs_manager.cc:359] Metadata directory not provided
I20260810 03:31:56.895879 30330 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:31:56.896224 30330 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:31:56.899540 30330 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786332336319544-28292-0/minicluster-data/master-0/data/instance:
uuid: "fa3f0f79c1034d2f8dc9edb8d61bcfc2"
format_stamp: "Formatted at 2026-08-10 03:31:56 on dist-test-slave-vpvm"
server_key: "46256868bf3f5d4dbdb6957c7b876b66"
server_key_iv: "02a41d0a49aeeac2a478fbf2807e949a"
server_key_version: "encryptionkey@0"
I20260810 03:31:56.900401 30330 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786332336319544-28292-0/minicluster-data/master-0/wal/instance:
uuid: "fa3f0f79c1034d2f8dc9edb8d61bcfc2"
format_stamp: "Formatted at 2026-08-10 03:31:56 on dist-test-slave-vpvm"
server_key: "46256868bf3f5d4dbdb6957c7b876b66"
server_key_iv: "02a41d0a49aeeac2a478fbf2807e949a"
server_key_version: "encryptionkey@0"
I20260810 03:31:56.906198 30330 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260810 03:31:56.909929 30353 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:31:56.911404 30330 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 03:31:56.911599 30330 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786332336319544-28292-0/minicluster-data/master-0/data,/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786332336319544-28292-0/minicluster-data/master-0/wal
uuid: "fa3f0f79c1034d2f8dc9edb8d61bcfc2"
format_stamp: "Formatted at 2026-08-10 03:31:56 on dist-test-slave-vpvm"
server_key: "46256868bf3f5d4dbdb6957c7b876b66"
server_key_iv: "02a41d0a49aeeac2a478fbf2807e949a"
server_key_version: "encryptionkey@0"
I20260810 03:31:56.911824 30330 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786332336319544-28292-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786332336319544-28292-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786332336319544-28292-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:31:56.931022 30330 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:31:56.935935 30330 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:31:56.936245 30330 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:31:56.955367 30330 rpc_server.cc:307] RPC server started. Bound to: 127.27.161.62:43211
I20260810 03:31:56.955389 30427 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.161.62:43211 every 8 connection(s)
I20260810 03:31:56.956835 30330 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786332336319544-28292-0/minicluster-data/master-0/data/info.pb
I20260810 03:31:56.958916 28292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iBH5/build/asan/bin/kudu as pid 30330
I20260810 03:31:56.959168 28292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786332336319544-28292-0/minicluster-data/master-0/wal/instance
I20260810 03:31:56.960013 28292 external_mini_cluster.cc:1428] Setting key 6c0f424295157767979cbf5651ad414c
I20260810 03:31:56.962064 30429 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:31:56.971330 30429 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.161.62" port: 43211 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 03:31:56 dist-test-slave-vpvm krb5kdc[30312](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332716, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.161.62@KRBTEST.COM for kudu/127.27.161.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.161.61@KRBTEST.COM; defaulting to no policy
I20260810 03:31:56.997298 30433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:31:56.975326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:43211 (local address 127.0.0.1:55524)
0810 03:31:56.976486 (+ 1160us) negotiation.cc:107] Waiting for socket to connect
0810 03:31:56.976540 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0810 03:31:56.978752 (+ 2212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:31:56.979727 (+ 975us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:31:56.979735 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:31:56.980501 (+ 766us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:31:56.981086 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:31:56.981114 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:31:56.982877 (+ 1763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:31:56.982883 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:31:56.984688 (+ 1805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:31:56.984704 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:31:56.985736 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:31:56.985743 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:31:56.985929 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:31:56.986664 (+ 735us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:31:56.986694 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:31:56.989679 (+ 2985us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:31:56.992936 (+ 3257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:31:56.992950 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:31:56.992964 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:31:56.993380 (+ 416us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:31:56.993794 (+ 414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:31:56.993799 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:31:56.993803 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:31:56.994022 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:31:56.994816 (+ 794us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:31:56.994824 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:31:56.995025 (+ 201us) client_negotiation.cc:769] Sending connection context
0810 03:31:56.995287 (+ 262us) client_negotiation.cc:240] Negotiation successful
0810 03:31:56.995916 (+ 629us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":848,"mutex_wait_us":136,"thread_start_us":601,"threads_started":1}
I20260810 03:31:56.997316 30437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:31:56.975326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55524 (local address 127.27.161.62:43211)
0810 03:31:56.977360 (+ 2034us) server_negotiation.cc:206] Beginning negotiation
0810 03:31:56.977369 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 03:31:56.977422 (+ 53us) server_negotiation.cc:407] Connection header received
0810 03:31:56.979202 (+ 1780us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:31:56.979209 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:31:56.979463 (+ 254us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:31:56.979643 (+ 180us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:31:56.981355 (+ 1712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:31:56.982689 (+ 1334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:31:56.984863 (+ 2174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:31:56.985467 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:31:56.985629 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:31:56.989932 (+ 4303us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:31:56.989941 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:31:56.989985 (+ 44us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:31:56.990030 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:31:56.992795 (+ 2765us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:31:56.993510 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:31:56.993517 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:31:56.993520 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:31:56.993685 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:31:56.994126 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:31:56.994131 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:31:56.994134 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:31:56.994672 (+ 538us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:31:56.994865 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0810 03:31:56.995457 (+ 592us) server_negotiation.cc:299] Negotiation successful
0810 03:31:56.995747 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1804,"thread_start_us":1578,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.161.61@KRBTEST.COM" created.
I20260810 03:31:57.000722 30429 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.161.61" port: 45273 } has no permanent_uuid. Determining permanent_uuid...
I20260810 03:31:57.001914 30433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:31:57.001408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.61:45273 (local address 127.0.0.1:49656)
0810 03:31:57.001656 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0810 03:31:57.001781 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.61:45273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
W20260810 03:31:57.002239 30354 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.161.61:45273: connect: Connection refused (error 111)
I20260810 03:31:57.004387 30433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:31:57.003890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.61:45273 (local address 127.0.0.1:49662)
0810 03:31:57.004072 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0810 03:31:57.004151 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.61:45273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260810 03:31:57.004630 30429 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.161.61:45273: Network error: Client connection negotiation failed: client connection to 127.27.161.61:45273: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.161.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.61.keytab.
Entry for principal kudu/127.27.161.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.61.keytab.
I20260810 03:31:57.043083 28292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iBH5/build/asan/bin/kudu
/tmp/dist-test-task24iBH5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786332336319544-28292-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786332336319544-28292-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786332336319544-28292-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786332336319544-28292-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.161.61:45273
--webserver_interface=127.27.161.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.61.keytab
--principal=kudu/127.27.161.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.161.20:37217
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.161.62:43211,127.27.161.61:45273,127.27.161.60:40293
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786332336319544-28292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786332336319544-28292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786332336319544-28292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 03:31:57.054072 30429 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.161.61" port: 45273 } attempt: 1
I20260810 03:31:57.055347 30433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:31:57.054877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.61:45273 (local address 127.0.0.1:49664)
0810 03:31:57.055080 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0810 03:31:57.055212 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.61:45273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260810 03:31:57.056313 30433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:31:57.055973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.61:45273 (local address 127.0.0.1:49672)
0810 03:31:57.056094 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0810 03:31:57.056173 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.61:45273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260810 03:31:57.056483 30429 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.161.61:45273: Network error: Client connection negotiation failed: client connection to 127.27.161.61:45273: connect: Connection refused (error 111)
I20260810 03:31:57.124755 30429 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.161.61" port: 45273 } attempt: 2
I20260810 03:31:57.126134 30433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:31:57.125511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.61:45273 (local address 127.0.0.1:49676)
0810 03:31:57.125793 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0810 03:31:57.125917 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.61:45273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260810 03:31:57.127329 30433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:31:57.126875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.61:45273 (local address 127.0.0.1:49680)
0810 03:31:57.127051 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0810 03:31:57.127142 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.61:45273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260810 03:31:57.127568 30429 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.161.61:45273: Network error: Client connection negotiation failed: client connection to 127.27.161.61:45273: connect: Connection refused (error 111)
I20260810 03:31:57.218863 30429 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.161.61" port: 45273 } attempt: 3
I20260810 03:31:57.220237 30433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:31:57.219670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.61:45273 (local address 127.0.0.1:49696)
0810 03:31:57.219924 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0810 03:31:57.220052 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.61:45273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260810 03:31:57.221474 30433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:31:57.221001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.61:45273 (local address 127.0.0.1:49708)
0810 03:31:57.221176 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0810 03:31:57.221273 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.61:45273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260810 03:31:57.221742 30429 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.161.61:45273: Network error: Client connection negotiation failed: client connection to 127.27.161.61:45273: connect: Connection refused (error 111)
I20260810 03:31:57.365060 30429 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.161.61" port: 45273 } attempt: 4
I20260810 03:31:57.366523 30433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:31:57.365846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.61:45273 (local address 127.0.0.1:49722)
0810 03:31:57.366216 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0810 03:31:57.366330 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.61:45273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196}
I20260810 03:31:57.367743 30433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:31:57.367272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.61:45273 (local address 127.0.0.1:49736)
0810 03:31:57.367435 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0810 03:31:57.367573 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.61:45273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260810 03:31:57.368006 30429 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.161.61:45273: Network error: Client connection negotiation failed: client connection to 127.27.161.61:45273: connect: Connection refused (error 111)
W20260810 03:31:57.401922 30443 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260810 03:31:57.402354 30443 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:31:57.402454 30443 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:31:57.402588 30443 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:31:57.410653 30443 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:31:57.410795 30443 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:31:57.410884 30443 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:31:57.410955 30443 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:31:57.411046 30443 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:31:57.421624 30443 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.161.20:37217
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786332336319544-28292-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786332336319544-28292-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.27.161.62:43211,127.27.161.61:45273,127.27.161.60:40293
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.61.keytab
--principal=kudu/127.27.161.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.161.61:45273
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786332336319544-28292-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.27.161.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786332336319544-28292-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:31:57.423038 30443 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:31:57.424456 30443 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:31:57.436319 30453 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:31:57.436298 30451 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:31:57.436651 30449 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:31:57.437052 30443 server_base.cc:1034] running on GCE node
I20260810 03:31:57.437860 30443 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:31:57.439399 30443 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:31:57.440629 30443 hybrid_clock.cc:648] HybridClock initialized: now 1786332717440557 us; error 47 us; skew 500 ppm
I20260810 03:31:57.440968 30443 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:31:57 dist-test-slave-vpvm krb5kdc[30312](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332717, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.161.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:31:57.444150 30443 init.cc:377] Logged in from keytab as kudu/127.27.161.61@KRBTEST.COM (short username kudu)
I20260810 03:31:57.445626 30443 webserver.cc:460] Webserver started at http://127.27.161.61:44597/ using document root <none> and password file <none>
I20260810 03:31:57.446379 30443 fs_manager.cc:359] Metadata directory not provided
I20260810 03:31:57.446481 30443 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:31:57.446815 30443 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:31:57.449970 30443 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786332336319544-28292-0/minicluster-data/master-1/data/instance:
uuid: "5db20d63e80f485d820b6cb2659ee542"
format_stamp: "Formatted at 2026-08-10 03:31:57 on dist-test-slave-vpvm"
server_key: "e73de422758a0da8f56b521c635e96cd"
server_key_iv: "01978ae19b0f94f39c802aae4417bf27"
server_key_version: "encryptionkey@0"
I20260810 03:31:57.450824 30443 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786332336319544-28292-0/minicluster-data/master-1/wal/instance:
uuid: "5db20d63e80f485d820b6cb2659ee542"
format_stamp: "Formatted at 2026-08-10 03:31:57 on dist-test-slave-vpvm"
server_key: "e73de422758a0da8f56b521c635e96cd"
server_key_iv: "01978ae19b0f94f39c802aae4417bf27"
server_key_version: "encryptionkey@0"
I20260810 03:31:57.456343 30443 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260810 03:31:57.459954 30460 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:31:57.461285 30443 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 03:31:57.461463 30443 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786332336319544-28292-0/minicluster-data/master-1/data,/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786332336319544-28292-0/minicluster-data/master-1/wal
uuid: "5db20d63e80f485d820b6cb2659ee542"
format_stamp: "Formatted at 2026-08-10 03:31:57 on dist-test-slave-vpvm"
server_key: "e73de422758a0da8f56b521c635e96cd"
server_key_iv: "01978ae19b0f94f39c802aae4417bf27"
server_key_version: "encryptionkey@0"
I20260810 03:31:57.461678 30443 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786332336319544-28292-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786332336319544-28292-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786332336319544-28292-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:31:57.479527 30443 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:31:57.483992 30443 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:31:57.484280 30443 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:31:57.502667 30443 rpc_server.cc:307] RPC server started. Bound to: 127.27.161.61:45273
I20260810 03:31:57.502674 30537 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.161.61:45273 every 8 connection(s)
I20260810 03:31:57.504192 30443 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786332336319544-28292-0/minicluster-data/master-1/data/info.pb
I20260810 03:31:57.508085 30539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:31:57.510016 28292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iBH5/build/asan/bin/kudu as pid 30443
I20260810 03:31:57.510233 28292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786332336319544-28292-0/minicluster-data/master-1/wal/instance
I20260810 03:31:57.510792 28292 external_mini_cluster.cc:1428] Setting key cd17ce085fa02782df4178364974bce7
I20260810 03:31:57.517417 30539 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.161.62" port: 43211 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 03:31:57 dist-test-slave-vpvm krb5kdc[30312](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332717, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.161.61@KRBTEST.COM for kudu/127.27.161.62@KRBTEST.COM
I20260810 03:31:57.541550 30541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:31:57.520803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55536 (local address 127.27.161.62:43211)
0810 03:31:57.521365 (+ 562us) server_negotiation.cc:206] Beginning negotiation
0810 03:31:57.521372 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:31:57.522481 (+ 1109us) server_negotiation.cc:407] Connection header received
0810 03:31:57.524545 (+ 2064us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:31:57.524551 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:31:57.524643 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:31:57.524774 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:31:57.526460 (+ 1686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:31:57.527492 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:31:57.529772 (+ 2280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:31:57.530232 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:31:57.530329 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:31:57.534603 (+ 4274us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:31:57.534611 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:31:57.534616 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:31:57.534676 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:31:57.537935 (+ 3259us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:31:57.538875 (+ 940us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:31:57.538881 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:31:57.538884 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:31:57.539042 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:31:57.539498 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:31:57.539503 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:31:57.539506 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:31:57.539905 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:31:57.540054 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0810 03:31:57.540941 (+ 887us) server_negotiation.cc:299] Negotiation successful
0810 03:31:57.541187 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":419,"thread_start_us":276,"threads_started":1}
I20260810 03:31:57.542486 30542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:31:57.521394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:43211 (local address 127.0.0.1:55536)
0810 03:31:57.522294 (+ 900us) negotiation.cc:107] Waiting for socket to connect
0810 03:31:57.522353 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0810 03:31:57.524247 (+ 1894us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:31:57.524983 (+ 736us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:31:57.524991 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:31:57.525577 (+ 586us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:31:57.526237 (+ 660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:31:57.526258 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:31:57.527726 (+ 1468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:31:57.527732 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:31:57.529614 (+ 1882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:31:57.529632 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:31:57.530476 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:31:57.530483 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:31:57.530774 (+ 291us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:31:57.531526 (+ 752us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:31:57.531564 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:31:57.534289 (+ 2725us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:31:57.538186 (+ 3897us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:31:57.538196 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:31:57.538213 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:31:57.538715 (+ 502us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:31:57.539145 (+ 430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:31:57.539148 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:31:57.539151 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:31:57.539358 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:31:57.540078 (+ 720us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:31:57.540086 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:31:57.540512 (+ 426us) client_negotiation.cc:769] Sending connection context
0810 03:31:57.540787 (+ 275us) client_negotiation.cc:240] Negotiation successful
0810 03:31:57.541232 (+ 445us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":584,"thread_start_us":344,"threads_started":1}
I20260810 03:31:57.544727 30539 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.161.61" port: 45273 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.27.161.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.161.60@KRBTEST.COM" created.
Aug 10 03:31:57 dist-test-slave-vpvm krb5kdc[30312](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332717, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.161.61@KRBTEST.COM for kudu/127.27.161.61@KRBTEST.COM
I20260810 03:31:57.560626 30542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:31:57.545449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.61:45273 (local address 127.0.0.1:49738)
0810 03:31:57.545676 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0810 03:31:57.545695 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 03:31:57.545814 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:31:57.547066 (+ 1252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:31:57.547070 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:31:57.547482 (+ 412us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:31:57.547741 (+ 259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:31:57.547752 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:31:57.549826 (+ 2074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:31:57.549830 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:31:57.550776 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:31:57.550790 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:31:57.551732 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:31:57.551737 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:31:57.551885 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:31:57.552431 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:31:57.552451 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:31:57.554746 (+ 2295us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:31:57.558067 (+ 3321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:31:57.558073 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:31:57.558078 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:31:57.558455 (+ 377us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:31:57.558849 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:31:57.558854 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:31:57.558858 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:31:57.558995 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:31:57.559632 (+ 637us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:31:57.559637 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:31:57.559771 (+ 134us) client_negotiation.cc:769] Sending connection context
0810 03:31:57.559889 (+ 118us) client_negotiation.cc:240] Negotiation successful
0810 03:31:57.560189 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":78}
I20260810 03:31:57.560670 30546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:31:57.545740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49738 (local address 127.27.161.61:45273)
0810 03:31:57.546434 (+ 694us) server_negotiation.cc:206] Beginning negotiation
0810 03:31:57.546443 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 03:31:57.546481 (+ 38us) server_negotiation.cc:407] Connection header received
0810 03:31:57.546554 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:31:57.546563 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:31:57.546862 (+ 299us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:31:57.547051 (+ 189us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:31:57.547951 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:31:57.549657 (+ 1706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:31:57.550931 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:31:57.551573 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:31:57.551684 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:31:57.554960 (+ 3276us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:31:57.554974 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:31:57.554991 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:31:57.555032 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:31:57.557865 (+ 2833us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:31:57.558607 (+ 742us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:31:57.558615 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:31:57.558617 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:31:57.558738 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:31:57.559097 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:31:57.559101 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:31:57.559103 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:31:57.559476 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:31:57.559685 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0810 03:31:57.560082 (+ 397us) server_negotiation.cc:299] Negotiation successful
0810 03:31:57.560319 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":490,"thread_start_us":328,"threads_started":1}
I20260810 03:31:57.563306 30539 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.161.60" port: 40293 } has no permanent_uuid. Determining permanent_uuid...
I20260810 03:31:57.565009 30542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:31:57.564368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.60:40293 (local address 127.0.0.1:46402)
0810 03:31:57.564589 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0810 03:31:57.564807 (+ 218us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.60:40293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260810 03:31:57.565428 30464 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.161.60:40293: connect: Connection refused (error 111)
I20260810 03:31:57.567273 30542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:31:57.566895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.60:40293 (local address 127.0.0.1:46418)
0810 03:31:57.567035 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0810 03:31:57.567136 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.60:40293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260810 03:31:57.567502 30539 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.161.60:40293: Network error: Client connection negotiation failed: client connection to 127.27.161.60:40293: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.161.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.60.keytab.
Entry for principal kudu/127.27.161.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.60.keytab.
I20260810 03:31:57.595111 28292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iBH5/build/asan/bin/kudu
/tmp/dist-test-task24iBH5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786332336319544-28292-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786332336319544-28292-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786332336319544-28292-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786332336319544-28292-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.161.60:40293
--webserver_interface=127.27.161.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.60.keytab
--principal=kudu/127.27.161.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.161.20:37217
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.161.62:43211,127.27.161.61:45273,127.27.161.60:40293
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786332336319544-28292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786332336319544-28292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786332336319544-28292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 03:31:57.616852 30539 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.161.60" port: 40293 } attempt: 1
I20260810 03:31:57.618274 30542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:31:57.617632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.60:40293 (local address 127.0.0.1:46422)
0810 03:31:57.617862 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0810 03:31:57.618073 (+ 211us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.60:40293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260810 03:31:57.619436 30542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:31:57.619041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.60:40293 (local address 127.0.0.1:46432)
0810 03:31:57.619207 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0810 03:31:57.619309 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.60:40293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260810 03:31:57.619736 30539 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.161.60:40293: Network error: Client connection negotiation failed: client connection to 127.27.161.60:40293: connect: Connection refused (error 111)
I20260810 03:31:57.667284 30429 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.161.61" port: 45273 } attempt: 5
Aug 10 03:31:57 dist-test-slave-vpvm krb5kdc[30312](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332716, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.161.62@KRBTEST.COM for kudu/127.27.161.61@KRBTEST.COM
I20260810 03:31:57.682050 30546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:31:57.668302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49750 (local address 127.27.161.61:45273)
0810 03:31:57.668472 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0810 03:31:57.668477 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:31:57.668497 (+ 20us) server_negotiation.cc:407] Connection header received
0810 03:31:57.668556 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:31:57.668560 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:31:57.668638 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:31:57.668735 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:31:57.669733 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:31:57.670472 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:31:57.672101 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:31:57.672544 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:31:57.672637 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:31:57.676475 (+ 3838us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:31:57.676481 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:31:57.676484 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:31:57.676520 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:31:57.679124 (+ 2604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:31:57.679998 (+ 874us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:31:57.680005 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:31:57.680009 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:31:57.680185 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:31:57.680645 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:31:57.680650 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:31:57.680652 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:31:57.680976 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:31:57.681108 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0810 03:31:57.681575 (+ 467us) server_negotiation.cc:299] Negotiation successful
0810 03:31:57.681776 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260810 03:31:57.682209 30433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:31:57.668042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.61:45273 (local address 127.0.0.1:49750)
0810 03:31:57.668278 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0810 03:31:57.668301 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0810 03:31:57.668440 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:31:57.668784 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:31:57.668789 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:31:57.669304 (+ 515us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:31:57.669567 (+ 263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:31:57.669582 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:31:57.670609 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:31:57.670615 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:31:57.671958 (+ 1343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:31:57.671975 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:31:57.672693 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:31:57.672700 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:31:57.672937 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:31:57.673598 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:31:57.673623 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:31:57.676242 (+ 2619us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:31:57.679294 (+ 3052us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:31:57.679300 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:31:57.679304 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:31:57.679851 (+ 547us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:31:57.680315 (+ 464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:31:57.680321 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:31:57.680324 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:31:57.680497 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:31:57.681104 (+ 607us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:31:57.681109 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:31:57.681309 (+ 200us) client_negotiation.cc:769] Sending connection context
0810 03:31:57.681474 (+ 165us) client_negotiation.cc:240] Negotiation successful
0810 03:31:57.681831 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":75}
I20260810 03:31:57.683365 30429 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.161.60" port: 40293 } has no permanent_uuid. Determining permanent_uuid...
I20260810 03:31:57.684468 30433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:31:57.684084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.60:40293 (local address 127.0.0.1:46434)
0810 03:31:57.684250 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0810 03:31:57.684327 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.60:40293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260810 03:31:57.685340 30433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:31:57.685049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.60:40293 (local address 127.0.0.1:46442)
0810 03:31:57.685172 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0810 03:31:57.685247 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.60:40293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260810 03:31:57.685534 30429 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.161.60:40293: Network error: Client connection negotiation failed: client connection to 127.27.161.60:40293: connect: Connection refused (error 111)
I20260810 03:31:57.687970 30539 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.161.60" port: 40293 } attempt: 2
I20260810 03:31:57.688944 30542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:31:57.688513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.60:40293 (local address 127.0.0.1:46450)
0810 03:31:57.688672 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0810 03:31:57.688778 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.60:40293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260810 03:31:57.689837 30542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:31:57.689505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.60:40293 (local address 127.0.0.1:46456)
0810 03:31:57.689645 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0810 03:31:57.689723 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.60:40293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260810 03:31:57.690071 30539 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.161.60:40293: Network error: Client connection negotiation failed: client connection to 127.27.161.60:40293: connect: Connection refused (error 111)
I20260810 03:31:57.736763 30429 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.161.60" port: 40293 } attempt: 1
I20260810 03:31:57.738018 30433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:31:57.737472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.60:40293 (local address 127.0.0.1:46462)
0810 03:31:57.737693 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0810 03:31:57.737790 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.60:40293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260810 03:31:57.738982 30433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:31:57.738656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.60:40293 (local address 127.0.0.1:46478)
0810 03:31:57.738801 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0810 03:31:57.738873 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.60:40293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260810 03:31:57.739181 30429 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.161.60:40293: Network error: Client connection negotiation failed: client connection to 127.27.161.60:40293: connect: Connection refused (error 111)
I20260810 03:31:57.781315 30539 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.161.60" port: 40293 } attempt: 3
I20260810 03:31:57.782521 30542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:31:57.782023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.60:40293 (local address 127.0.0.1:46494)
0810 03:31:57.782239 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0810 03:31:57.782334 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.60:40293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260810 03:31:57.783581 30542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:31:57.783142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.60:40293 (local address 127.0.0.1:46500)
0810 03:31:57.783386 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0810 03:31:57.783463 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.60:40293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142}
W20260810 03:31:57.783777 30539 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.161.60:40293: Network error: Client connection negotiation failed: client connection to 127.27.161.60:40293: connect: Connection refused (error 111)
I20260810 03:31:57.807379 30429 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.161.60" port: 40293 } attempt: 2
I20260810 03:31:57.808552 30433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:31:57.808043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.60:40293 (local address 127.0.0.1:46512)
0810 03:31:57.808288 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0810 03:31:57.808382 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.60:40293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260810 03:31:57.809495 30433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:31:57.809159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.60:40293 (local address 127.0.0.1:46528)
0810 03:31:57.809314 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0810 03:31:57.809385 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.60:40293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260810 03:31:57.809707 30429 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.161.60:40293: Network error: Client connection negotiation failed: client connection to 127.27.161.60:40293: connect: Connection refused (error 111)
I20260810 03:31:57.915935 30429 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.161.60" port: 40293 } attempt: 3
I20260810 03:31:57.917001 30433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:31:57.916544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.60:40293 (local address 127.0.0.1:46542)
0810 03:31:57.916740 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0810 03:31:57.916828 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.60:40293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260810 03:31:57.918258 30433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:31:57.917826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.60:40293 (local address 127.0.0.1:46556)
0810 03:31:57.918060 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0810 03:31:57.918144 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.60:40293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
W20260810 03:31:57.918470 30429 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.161.60:40293: Network error: Client connection negotiation failed: client connection to 127.27.161.60:40293: connect: Connection refused (error 111)
I20260810 03:31:57.926996 30539 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.161.60" port: 40293 } attempt: 4
I20260810 03:31:57.928126 30542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:31:57.927650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.60:40293 (local address 127.0.0.1:46570)
0810 03:31:57.927876 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0810 03:31:57.927964 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.60:40293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260810 03:31:57.929080 30542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:31:57.928741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.60:40293 (local address 127.0.0.1:46584)
0810 03:31:57.928893 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0810 03:31:57.928963 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.60:40293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260810 03:31:57.929312 30539 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.161.60:40293: Network error: Client connection negotiation failed: client connection to 127.27.161.60:40293: connect: Connection refused (error 111)
W20260810 03:31:57.965312 30550 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260810 03:31:57.965651 30550 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:31:57.965752 30550 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:31:57.965863 30550 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:31:57.973628 30550 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:31:57.973743 30550 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:31:57.973836 30550 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:31:57.973901 30550 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:31:57.974010 30550 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:31:57.984681 30550 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.161.20:37217
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786332336319544-28292-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786332336319544-28292-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.27.161.62:43211,127.27.161.61:45273,127.27.161.60:40293
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.60.keytab
--principal=kudu/127.27.161.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.161.60:40293
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786332336319544-28292-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.27.161.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786332336319544-28292-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:31:57.986140 30550 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:31:57.987567 30550 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:31:57.999475 30557 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:31:57.999389 30558 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:31:57.999431 30561 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:31:58.000340 30550 server_base.cc:1034] running on GCE node
I20260810 03:31:58.001166 30550 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:31:58.002724 30550 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:31:58.003997 30550 hybrid_clock.cc:648] HybridClock initialized: now 1786332718003926 us; error 44 us; skew 500 ppm
I20260810 03:31:58.004355 30550 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:31:58 dist-test-slave-vpvm krb5kdc[30312](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332718, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.161.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:31:58.007921 30550 init.cc:377] Logged in from keytab as kudu/127.27.161.60@KRBTEST.COM (short username kudu)
I20260810 03:31:58.009526 30550 webserver.cc:460] Webserver started at http://127.27.161.60:36345/ using document root <none> and password file <none>
I20260810 03:31:58.010316 30550 fs_manager.cc:359] Metadata directory not provided
I20260810 03:31:58.010428 30550 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:31:58.010762 30550 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:31:58.013849 30550 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786332336319544-28292-0/minicluster-data/master-2/data/instance:
uuid: "da7db9c2093848b9b767520fb1bf18f9"
format_stamp: "Formatted at 2026-08-10 03:31:58 on dist-test-slave-vpvm"
server_key: "8e02cb395fe947cafb1e7a1d0d63119d"
server_key_iv: "7739ef5cbb6ff6546c26fa2725c55eb9"
server_key_version: "encryptionkey@0"
I20260810 03:31:58.014719 30550 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786332336319544-28292-0/minicluster-data/master-2/wal/instance:
uuid: "da7db9c2093848b9b767520fb1bf18f9"
format_stamp: "Formatted at 2026-08-10 03:31:58 on dist-test-slave-vpvm"
server_key: "8e02cb395fe947cafb1e7a1d0d63119d"
server_key_iv: "7739ef5cbb6ff6546c26fa2725c55eb9"
server_key_version: "encryptionkey@0"
I20260810 03:31:58.020131 30550 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260810 03:31:58.023803 30571 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:31:58.025239 30550 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260810 03:31:58.025430 30550 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786332336319544-28292-0/minicluster-data/master-2/data,/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786332336319544-28292-0/minicluster-data/master-2/wal
uuid: "da7db9c2093848b9b767520fb1bf18f9"
format_stamp: "Formatted at 2026-08-10 03:31:58 on dist-test-slave-vpvm"
server_key: "8e02cb395fe947cafb1e7a1d0d63119d"
server_key_iv: "7739ef5cbb6ff6546c26fa2725c55eb9"
server_key_version: "encryptionkey@0"
I20260810 03:31:58.025663 30550 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786332336319544-28292-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786332336319544-28292-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786332336319544-28292-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:31:58.043597 30550 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:31:58.048033 30550 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:31:58.048321 30550 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:31:58.066900 30550 rpc_server.cc:307] RPC server started. Bound to: 127.27.161.60:40293
I20260810 03:31:58.066870 30648 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.161.60:40293 every 8 connection(s)
I20260810 03:31:58.068338 30550 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786332336319544-28292-0/minicluster-data/master-2/data/info.pb
I20260810 03:31:58.072290 30650 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:31:58.075369 28292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iBH5/build/asan/bin/kudu as pid 30550
I20260810 03:31:58.075580 28292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786332336319544-28292-0/minicluster-data/master-2/wal/instance
I20260810 03:31:58.076038 28292 external_mini_cluster.cc:1428] Setting key a428e11375c36de0d134503727493bb7
I20260810 03:31:58.080708 30650 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.161.62" port: 43211 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 03:31:58 dist-test-slave-vpvm krb5kdc[30312](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332716, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.161.60@KRBTEST.COM
Aug 10 03:31:58 dist-test-slave-vpvm krb5kdc[30312](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332718, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.161.60@KRBTEST.COM for kudu/127.27.161.62@KRBTEST.COM
I20260810 03:31:58.095705 30429 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.161.60" port: 40293 } attempt: 4
I20260810 03:31:58.099655 30658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:31:58.084299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55548 (local address 127.27.161.62:43211)
0810 03:31:58.085539 (+ 1240us) server_negotiation.cc:206] Beginning negotiation
0810 03:31:58.085544 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:31:58.085559 (+ 15us) server_negotiation.cc:407] Connection header received
0810 03:31:58.085609 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:31:58.085613 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:31:58.085679 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:31:58.085777 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:31:58.086975 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:31:58.087674 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:31:58.089170 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:31:58.089574 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:31:58.089695 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:31:58.093307 (+ 3612us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:31:58.093312 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:31:58.093315 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:31:58.093364 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:31:58.096128 (+ 2764us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:31:58.097095 (+ 967us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:31:58.097101 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:31:58.097104 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:31:58.097266 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:31:58.097767 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:31:58.097773 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:31:58.097776 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:31:58.098125 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:31:58.098273 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0810 03:31:58.099056 (+ 783us) server_negotiation.cc:299] Negotiation successful
0810 03:31:58.099267 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1138,"thread_start_us":1048,"threads_started":1}
I20260810 03:31:58.100443 30656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:31:58.078160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46598 (local address 127.27.161.60:40293)
0810 03:31:58.078972 (+ 812us) server_negotiation.cc:206] Beginning negotiation
0810 03:31:58.079000 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0810 03:31:58.079055 (+ 55us) server_negotiation.cc:407] Connection header received
0810 03:31:58.081341 (+ 2286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:31:58.081382 (+ 41us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:31:58.082013 (+ 631us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:31:58.082512 (+ 499us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:31:58.083995 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:31:58.085878 (+ 1883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:31:58.087182 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:31:58.087831 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:31:58.088034 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:31:58.092129 (+ 4095us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:31:58.092138 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:31:58.092151 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:31:58.092191 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:31:58.095014 (+ 2823us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:31:58.095666 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:31:58.095673 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:31:58.095681 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:31:58.095847 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:31:58.096282 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:31:58.096288 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:31:58.096291 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:31:58.097167 (+ 876us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:31:58.097495 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0810 03:31:58.098585 (+ 1090us) server_negotiation.cc:299] Negotiation successful
0810 03:31:58.098995 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":559,"thread_start_us":327,"threads_started":1}
I20260810 03:31:58.100488 30657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:31:58.084223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:43211 (local address 127.0.0.1:55548)
0810 03:31:58.084952 (+ 729us) negotiation.cc:107] Waiting for socket to connect
0810 03:31:58.084999 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0810 03:31:58.085161 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:31:58.085878 (+ 717us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:31:58.085885 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:31:58.086571 (+ 686us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:31:58.086824 (+ 253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:31:58.086833 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:31:58.087834 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:31:58.087838 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:31:58.089017 (+ 1179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:31:58.089030 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:31:58.089816 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:31:58.089824 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:31:58.089991 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:31:58.090600 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:31:58.090634 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:31:58.093107 (+ 2473us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:31:58.096250 (+ 3143us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:31:58.096262 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:31:58.096271 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:31:58.096941 (+ 670us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:31:58.097426 (+ 485us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:31:58.097430 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:31:58.097433 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:31:58.097621 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:31:58.098395 (+ 774us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:31:58.098401 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:31:58.098691 (+ 290us) client_negotiation.cc:769] Sending connection context
0810 03:31:58.098924 (+ 233us) client_negotiation.cc:240] Negotiation successful
0810 03:31:58.099244 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":470,"spinlock_wait_cycles":23424,"thread_start_us":298,"threads_started":1}
I20260810 03:31:58.103302 30650 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.161.61" port: 45273 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 03:31:58 dist-test-slave-vpvm krb5kdc[30312](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332716, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.161.62@KRBTEST.COM for kudu/127.27.161.60@KRBTEST.COM
Aug 10 03:31:58 dist-test-slave-vpvm krb5kdc[30312](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332718, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.161.60@KRBTEST.COM for kudu/127.27.161.61@KRBTEST.COM
I20260810 03:31:58.113052 30433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:31:58.096331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.60:40293 (local address 127.0.0.1:46606)
0810 03:31:58.096517 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0810 03:31:58.096536 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 03:31:58.096644 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:31:58.097666 (+ 1022us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:31:58.097671 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:31:58.098168 (+ 497us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:31:58.098388 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:31:58.098401 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:31:58.100098 (+ 1697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:31:58.100104 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:31:58.101468 (+ 1364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:31:58.101485 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:31:58.102623 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:31:58.102630 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:31:58.102830 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:31:58.103463 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:31:58.103497 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:31:58.106113 (+ 2616us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:31:58.110216 (+ 4103us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:31:58.110221 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:31:58.110225 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:31:58.110716 (+ 491us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:31:58.111182 (+ 466us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:31:58.111188 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:31:58.111192 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:31:58.111382 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:31:58.112034 (+ 652us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:31:58.112039 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:31:58.112193 (+ 154us) client_negotiation.cc:769] Sending connection context
0810 03:31:58.112350 (+ 157us) client_negotiation.cc:240] Negotiation successful
0810 03:31:58.112640 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260810 03:31:58.113085 30660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:31:58.096709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46606 (local address 127.27.161.60:40293)
0810 03:31:58.097361 (+ 652us) server_negotiation.cc:206] Beginning negotiation
0810 03:31:58.097367 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:31:58.097392 (+ 25us) server_negotiation.cc:407] Connection header received
0810 03:31:58.097431 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:31:58.097435 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:31:58.097502 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:31:58.097613 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:31:58.098561 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:31:58.099923 (+ 1362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:31:58.101664 (+ 1741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:31:58.102432 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:31:58.102598 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:31:58.106348 (+ 3750us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:31:58.106356 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:31:58.106361 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:31:58.106399 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:31:58.110043 (+ 3644us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:31:58.110887 (+ 844us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:31:58.110892 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:31:58.110896 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:31:58.111049 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:31:58.111506 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:31:58.111511 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:31:58.111514 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:31:58.111895 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:31:58.112066 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0810 03:31:58.112465 (+ 399us) server_negotiation.cc:299] Negotiation successful
0810 03:31:58.112696 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":520,"thread_start_us":292,"threads_started":1}
Aug 10 03:31:58 dist-test-slave-vpvm krb5kdc[30312](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332716, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.161.62@KRBTEST.COM
I20260810 03:31:58.118062 30657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:31:58.104370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.61:45273 (local address 127.0.0.1:49766)
0810 03:31:58.104530 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0810 03:31:58.104547 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 03:31:58.104636 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:31:58.105231 (+ 595us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:31:58.105236 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:31:58.105708 (+ 472us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:31:58.105889 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:31:58.105900 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:31:58.107005 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:31:58.107011 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:31:58.107932 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:31:58.107947 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:31:58.108761 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:31:58.108767 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:31:58.108979 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:31:58.109553 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:31:58.109576 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:31:58.111923 (+ 2347us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:31:58.115522 (+ 3599us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:31:58.115527 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:31:58.115531 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:31:58.116021 (+ 490us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:31:58.116441 (+ 420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:31:58.116447 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:31:58.116451 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:31:58.116595 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:31:58.117092 (+ 497us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:31:58.117097 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:31:58.117247 (+ 150us) client_negotiation.cc:769] Sending connection context
0810 03:31:58.117384 (+ 137us) client_negotiation.cc:240] Negotiation successful
0810 03:31:58.117671 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260810 03:31:58.118088 30546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:31:58.104750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49766 (local address 127.27.161.61:45273)
0810 03:31:58.104917 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0810 03:31:58.104923 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:31:58.104940 (+ 17us) server_negotiation.cc:407] Connection header received
0810 03:31:58.104997 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:31:58.105002 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:31:58.105076 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:31:58.105168 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:31:58.106163 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:31:58.106862 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:31:58.108067 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:31:58.108612 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:31:58.108726 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:31:58.112143 (+ 3417us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:31:58.112150 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:31:58.112154 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:31:58.112205 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:31:58.115400 (+ 3195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:31:58.116145 (+ 745us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:31:58.116151 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:31:58.116154 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:31:58.116327 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:31:58.116698 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:31:58.116703 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:31:58.116707 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:31:58.116969 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:31:58.117112 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0810 03:31:58.117495 (+ 383us) server_negotiation.cc:299] Negotiation successful
0810 03:31:58.117699 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260810 03:31:58.119463 30650 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.161.60" port: 40293 } has no permanent_uuid. Determining permanent_uuid...
I20260810 03:31:58.120924 30429 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fa3f0f79c1034d2f8dc9edb8d61bcfc2: Bootstrap starting.
I20260810 03:31:58.121140 30658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:31:58.103906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55560 (local address 127.27.161.62:43211)
0810 03:31:58.104107 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0810 03:31:58.104113 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:31:58.104130 (+ 17us) server_negotiation.cc:407] Connection header received
0810 03:31:58.104309 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:31:58.104314 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:31:58.104410 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:31:58.104520 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:31:58.106299 (+ 1779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:31:58.107306 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:31:58.109003 (+ 1697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:31:58.109575 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:31:58.109705 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:31:58.114622 (+ 4917us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:31:58.114627 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:31:58.114630 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:31:58.114668 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:31:58.118143 (+ 3475us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:31:58.118972 (+ 829us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:31:58.118977 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:31:58.118980 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:31:58.119129 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:31:58.119582 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:31:58.119586 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:31:58.119588 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:31:58.119882 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:31:58.120166 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0810 03:31:58.120481 (+ 315us) server_negotiation.cc:299] Negotiation successful
0810 03:31:58.120714 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260810 03:31:58.124929 30429 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fa3f0f79c1034d2f8dc9edb8d61bcfc2: Neither blocks nor log segments found. Creating new log.
I20260810 03:31:58.126310 30429 log.cc:824] T 00000000000000000000000000000000 P fa3f0f79c1034d2f8dc9edb8d61bcfc2: Log is configured to *not* fsync() on all Append() calls
Aug 10 03:31:58 dist-test-slave-vpvm krb5kdc[30312](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332718, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.161.60@KRBTEST.COM for kudu/127.27.161.60@KRBTEST.COM
I20260810 03:31:58.129083 30429 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fa3f0f79c1034d2f8dc9edb8d61bcfc2: No bootstrap required, opened a new log
Aug 10 03:31:58 dist-test-slave-vpvm krb5kdc[30312](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332716, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.161.61@KRBTEST.COM
I20260810 03:31:58.134929 30657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:31:58.120347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.60:40293 (local address 127.0.0.1:46620)
0810 03:31:58.120901 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0810 03:31:58.120920 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 03:31:58.121026 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:31:58.121462 (+ 436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:31:58.121466 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:31:58.121923 (+ 457us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:31:58.122142 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:31:58.122155 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:31:58.123457 (+ 1302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:31:58.123462 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:31:58.124768 (+ 1306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:31:58.124781 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:31:58.125477 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:31:58.125482 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:31:58.125660 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:31:58.126330 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:31:58.126352 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:31:58.129073 (+ 2721us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:31:58.132482 (+ 3409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:31:58.132488 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:31:58.132492 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:31:58.132888 (+ 396us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:31:58.133223 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:31:58.133228 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:31:58.133232 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:31:58.133356 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:31:58.133930 (+ 574us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:31:58.133935 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:31:58.134098 (+ 163us) client_negotiation.cc:769] Sending connection context
0810 03:31:58.134256 (+ 158us) client_negotiation.cc:240] Negotiation successful
0810 03:31:58.134558 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":477}
I20260810 03:31:58.135867 30660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:31:58.120652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46620 (local address 127.27.161.60:40293)
0810 03:31:58.120808 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0810 03:31:58.120813 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:31:58.121122 (+ 309us) server_negotiation.cc:407] Connection header received
0810 03:31:58.121160 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:31:58.121164 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:31:58.121240 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:31:58.121320 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:31:58.122286 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:31:58.123319 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:31:58.124891 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:31:58.125348 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:31:58.125455 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:31:58.129260 (+ 3805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:31:58.129266 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:31:58.129270 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:31:58.129304 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:31:58.132368 (+ 3064us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:31:58.132991 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:31:58.132996 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:31:58.133000 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:31:58.133122 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:31:58.133451 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:31:58.133456 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:31:58.133460 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:31:58.133746 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:31:58.133882 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0810 03:31:58.135338 (+ 1456us) server_negotiation.cc:299] Negotiation successful
0810 03:31:58.135546 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260810 03:31:58.139544 30546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:31:58.122643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49778 (local address 127.27.161.61:45273)
0810 03:31:58.122808 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0810 03:31:58.122814 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:31:58.122871 (+ 57us) server_negotiation.cc:407] Connection header received
0810 03:31:58.123031 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:31:58.123037 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:31:58.123097 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:31:58.123172 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:31:58.124722 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:31:58.125749 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:31:58.127403 (+ 1654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:31:58.127980 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:31:58.128078 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:31:58.133799 (+ 5721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:31:58.133806 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:31:58.133810 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:31:58.133849 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:31:58.137046 (+ 3197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:31:58.137663 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:31:58.137669 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:31:58.137672 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:31:58.137812 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:31:58.138201 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:31:58.138206 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:31:58.138209 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:31:58.138478 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:31:58.138718 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0810 03:31:58.139034 (+ 316us) server_negotiation.cc:299] Negotiation successful
0810 03:31:58.139266 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260810 03:31:58.140515 30429 raft_consensus.cc:348] T 00000000000000000000000000000000 P fa3f0f79c1034d2f8dc9edb8d61bcfc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa3f0f79c1034d2f8dc9edb8d61bcfc2" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 43211 } } peers { permanent_uuid: "5db20d63e80f485d820b6cb2659ee542" member_type: VOTER last_known_addr { host: "127.27.161.61" port: 45273 } } peers { permanent_uuid: "da7db9c2093848b9b767520fb1bf18f9" member_type: VOTER last_known_addr { host: "127.27.161.60" port: 40293 } }
I20260810 03:31:58.140935 30429 raft_consensus.cc:729] T 00000000000000000000000000000000 P fa3f0f79c1034d2f8dc9edb8d61bcfc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa3f0f79c1034d2f8dc9edb8d61bcfc2, State: Initialized, Role: FOLLOWER
I20260810 03:31:58.141135 30650 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da7db9c2093848b9b767520fb1bf18f9: Bootstrap starting.
I20260810 03:31:58.141466 30429 consensus_queue.cc:260] T 00000000000000000000000000000000 P fa3f0f79c1034d2f8dc9edb8d61bcfc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa3f0f79c1034d2f8dc9edb8d61bcfc2" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 43211 } } peers { permanent_uuid: "5db20d63e80f485d820b6cb2659ee542" member_type: VOTER last_known_addr { host: "127.27.161.61" port: 45273 } } peers { permanent_uuid: "da7db9c2093848b9b767520fb1bf18f9" member_type: VOTER last_known_addr { host: "127.27.161.60" port: 40293 } }
I20260810 03:31:58.142545 30662 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa3f0f79c1034d2f8dc9edb8d61bcfc2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa3f0f79c1034d2f8dc9edb8d61bcfc2" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 43211 } } peers { permanent_uuid: "5db20d63e80f485d820b6cb2659ee542" member_type: VOTER last_known_addr { host: "127.27.161.61" port: 45273 } } peers { permanent_uuid: "da7db9c2093848b9b767520fb1bf18f9" member_type: VOTER last_known_addr { host: "127.27.161.60" port: 40293 } } }
I20260810 03:31:58.143056 30662 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa3f0f79c1034d2f8dc9edb8d61bcfc2 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 03:31:58.143855 30429 sys_catalog.cc:564] T 00000000000000000000000000000000 P fa3f0f79c1034d2f8dc9edb8d61bcfc2 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:31:58.144224 28292 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 03:31:58.146795 30388 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "fa3f0f79c1034d2f8dc9edb8d61bcfc2"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55560
I20260810 03:31:58.147220 30388 raft_consensus.cc:482] T 00000000000000000000000000000000 P fa3f0f79c1034d2f8dc9edb8d61bcfc2 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260810 03:31:58.147411 30388 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fa3f0f79c1034d2f8dc9edb8d61bcfc2 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:31:58.147500 30650 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P da7db9c2093848b9b767520fb1bf18f9: Neither blocks nor log segments found. Creating new log.
I20260810 03:31:58.148954 30650 log.cc:824] T 00000000000000000000000000000000 P da7db9c2093848b9b767520fb1bf18f9: Log is configured to *not* fsync() on all Append() calls
I20260810 03:31:58.149936 30388 raft_consensus.cc:504] T 00000000000000000000000000000000 P fa3f0f79c1034d2f8dc9edb8d61bcfc2 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa3f0f79c1034d2f8dc9edb8d61bcfc2" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 43211 } } peers { permanent_uuid: "5db20d63e80f485d820b6cb2659ee542" member_type: VOTER last_known_addr { host: "127.27.161.61" port: 45273 } } peers { permanent_uuid: "da7db9c2093848b9b767520fb1bf18f9" member_type: VOTER last_known_addr { host: "127.27.161.60" port: 40293 } }
I20260810 03:31:58.152580 30388 leader_election.cc:288] T 00000000000000000000000000000000 P fa3f0f79c1034d2f8dc9edb8d61bcfc2 [CANDIDATE]: Term 1 election: Requested vote from peers 5db20d63e80f485d820b6cb2659ee542 (127.27.161.61:45273), da7db9c2093848b9b767520fb1bf18f9 (127.27.161.60:40293)
I20260810 03:31:58.152772 30650 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da7db9c2093848b9b767520fb1bf18f9: No bootstrap required, opened a new log
I20260810 03:31:58.165582 30650 raft_consensus.cc:348] T 00000000000000000000000000000000 P da7db9c2093848b9b767520fb1bf18f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa3f0f79c1034d2f8dc9edb8d61bcfc2" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 43211 } } peers { permanent_uuid: "5db20d63e80f485d820b6cb2659ee542" member_type: VOTER last_known_addr { host: "127.27.161.61" port: 45273 } } peers { permanent_uuid: "da7db9c2093848b9b767520fb1bf18f9" member_type: VOTER last_known_addr { host: "127.27.161.60" port: 40293 } }
I20260810 03:31:58.166190 30650 raft_consensus.cc:729] T 00000000000000000000000000000000 P da7db9c2093848b9b767520fb1bf18f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da7db9c2093848b9b767520fb1bf18f9, State: Initialized, Role: FOLLOWER
I20260810 03:31:58.166818 30650 consensus_queue.cc:260] T 00000000000000000000000000000000 P da7db9c2093848b9b767520fb1bf18f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa3f0f79c1034d2f8dc9edb8d61bcfc2" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 43211 } } peers { permanent_uuid: "5db20d63e80f485d820b6cb2659ee542" member_type: VOTER last_known_addr { host: "127.27.161.61" port: 45273 } } peers { permanent_uuid: "da7db9c2093848b9b767520fb1bf18f9" member_type: VOTER last_known_addr { host: "127.27.161.60" port: 40293 } }
I20260810 03:31:58.168509 30671 sys_catalog.cc:455] T 00000000000000000000000000000000 P da7db9c2093848b9b767520fb1bf18f9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa3f0f79c1034d2f8dc9edb8d61bcfc2" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 43211 } } peers { permanent_uuid: "5db20d63e80f485d820b6cb2659ee542" member_type: VOTER last_known_addr { host: "127.27.161.61" port: 45273 } } peers { permanent_uuid: "da7db9c2093848b9b767520fb1bf18f9" member_type: VOTER last_known_addr { host: "127.27.161.60" port: 40293 } } }
I20260810 03:31:58.169569 30671 sys_catalog.cc:458] T 00000000000000000000000000000000 P da7db9c2093848b9b767520fb1bf18f9 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 03:31:58.169798 30650 sys_catalog.cc:564] T 00000000000000000000000000000000 P da7db9c2093848b9b767520fb1bf18f9 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:31:58.169875 30609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fa3f0f79c1034d2f8dc9edb8d61bcfc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "da7db9c2093848b9b767520fb1bf18f9"
I20260810 03:31:58.170454 30609 raft_consensus.cc:3037] T 00000000000000000000000000000000 P da7db9c2093848b9b767520fb1bf18f9 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:31:58.174429 30609 raft_consensus.cc:2454] T 00000000000000000000000000000000 P da7db9c2093848b9b767520fb1bf18f9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa3f0f79c1034d2f8dc9edb8d61bcfc2 in term 1.
I20260810 03:31:58.175374 30359 leader_election.cc:302] T 00000000000000000000000000000000 P fa3f0f79c1034d2f8dc9edb8d61bcfc2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: da7db9c2093848b9b767520fb1bf18f9, fa3f0f79c1034d2f8dc9edb8d61bcfc2; no voters:
I20260810 03:31:58.175927 30662 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fa3f0f79c1034d2f8dc9edb8d61bcfc2 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:31:58.178252 30662 raft_consensus.cc:686] T 00000000000000000000000000000000 P fa3f0f79c1034d2f8dc9edb8d61bcfc2 [term 1 LEADER]: Becoming Leader. State: Replica: fa3f0f79c1034d2f8dc9edb8d61bcfc2, State: Running, Role: LEADER
I20260810 03:31:58.178943 30662 consensus_queue.cc:237] T 00000000000000000000000000000000 P fa3f0f79c1034d2f8dc9edb8d61bcfc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa3f0f79c1034d2f8dc9edb8d61bcfc2" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 43211 } } peers { permanent_uuid: "5db20d63e80f485d820b6cb2659ee542" member_type: VOTER last_known_addr { host: "127.27.161.61" port: 45273 } } peers { permanent_uuid: "da7db9c2093848b9b767520fb1bf18f9" member_type: VOTER last_known_addr { host: "127.27.161.60" port: 40293 } }
W20260810 03:31:58.189419 30362 tablet.cc:2369] T 00000000000000000000000000000000 P fa3f0f79c1034d2f8dc9edb8d61bcfc2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 03:31:58.209059 30684 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa3f0f79c1034d2f8dc9edb8d61bcfc2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fa3f0f79c1034d2f8dc9edb8d61bcfc2. Latest consensus state: current_term: 1 leader_uuid: "fa3f0f79c1034d2f8dc9edb8d61bcfc2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa3f0f79c1034d2f8dc9edb8d61bcfc2" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 43211 } } peers { permanent_uuid: "5db20d63e80f485d820b6cb2659ee542" member_type: VOTER last_known_addr { host: "127.27.161.61" port: 45273 } } peers { permanent_uuid: "da7db9c2093848b9b767520fb1bf18f9" member_type: VOTER last_known_addr { host: "127.27.161.60" port: 40293 } } }
I20260810 03:31:58.209548 30684 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa3f0f79c1034d2f8dc9edb8d61bcfc2 [sys.catalog]: This master's current role is: LEADER
I20260810 03:31:58.214056 30707 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 03:31:58.228519 30539 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.161.60" port: 40293 } attempt: 5
I20260810 03:31:58.234233 30713 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260810 03:31:58.220654 30330 master_runner.cc:186] Error getting master registration for 127.27.161.61:45273: OK, instance_id { permanent_uuid: "5db20d63e80f485d820b6cb2659ee542" instance_seqno: 1786332717497264 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
Aug 10 03:31:58 dist-test-slave-vpvm krb5kdc[30312](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332717, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.161.61@KRBTEST.COM for kudu/127.27.161.60@KRBTEST.COM
I20260810 03:31:58.239688 30330 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 03:31:58.239781 30707 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 03:31:58.243115 30550 master_runner.cc:186] Error getting master registration for 127.27.161.61:45273: OK, instance_id { permanent_uuid: "5db20d63e80f485d820b6cb2659ee542" instance_seqno: 1786332717497264 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260810 03:31:58.257225 30550 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 03:31:58.259336 30546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:31:58.203973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49786 (local address 127.27.161.61:45273)
0810 03:31:58.204203 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0810 03:31:58.204210 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:31:58.210418 (+ 6208us) server_negotiation.cc:407] Connection header received
0810 03:31:58.210728 (+ 310us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:31:58.210735 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:31:58.210845 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:31:58.210963 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:31:58.213370 (+ 2407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:31:58.214548 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:31:58.232282 (+ 17734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:31:58.232993 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:31:58.233085 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:31:58.240410 (+ 7325us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:31:58.240417 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:31:58.240422 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:31:58.240462 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:31:58.243900 (+ 3438us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:31:58.244884 (+ 984us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:31:58.244891 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:31:58.244895 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:31:58.245069 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:31:58.245528 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:31:58.245534 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:31:58.245538 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:31:58.245964 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:31:58.246235 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0810 03:31:58.247621 (+ 1386us) server_negotiation.cc:299] Negotiation successful
0810 03:31:58.247895 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":40,"server-negotiator.queue_time_us":87}
I20260810 03:31:58.262339 30542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:31:58.229295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.60:40293 (local address 127.0.0.1:46644)
0810 03:31:58.229717 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0810 03:31:58.229759 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0810 03:31:58.230099 (+ 340us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:31:58.230867 (+ 768us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:31:58.230874 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:31:58.231456 (+ 582us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:31:58.231712 (+ 256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:31:58.231727 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:31:58.233091 (+ 1364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:31:58.233098 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:31:58.234549 (+ 1451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:31:58.234565 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:31:58.235779 (+ 1214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:31:58.235785 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:31:58.235988 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:31:58.236906 (+ 918us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:31:58.236930 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:31:58.239657 (+ 2727us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:31:58.254145 (+ 14488us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:31:58.254152 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:31:58.254156 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:31:58.259662 (+ 5506us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:31:58.260261 (+ 599us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:31:58.260267 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:31:58.260271 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:31:58.260479 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:31:58.261235 (+ 756us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:31:58.261241 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:31:58.261426 (+ 185us) client_negotiation.cc:769] Sending connection context
0810 03:31:58.261572 (+ 146us) client_negotiation.cc:240] Negotiation successful
0810 03:31:58.261916 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65,"mutex_wait_us":39}
I20260810 03:31:58.263588 30656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:31:58.229333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46644 (local address 127.27.161.60:40293)
0810 03:31:58.229525 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0810 03:31:58.229532 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:31:58.230290 (+ 758us) server_negotiation.cc:407] Connection header received
0810 03:31:58.230391 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:31:58.230404 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:31:58.230519 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:31:58.230615 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:31:58.231894 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:31:58.232951 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:31:58.234777 (+ 1826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:31:58.235440 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:31:58.235560 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:31:58.250206 (+ 14646us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:31:58.250214 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:31:58.250220 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:31:58.250270 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:31:58.253917 (+ 3647us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:31:58.259876 (+ 5959us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:31:58.259883 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:31:58.259886 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:31:58.260094 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:31:58.260651 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:31:58.260656 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:31:58.260659 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:31:58.261088 (+ 429us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:31:58.262829 (+ 1741us) server_negotiation.cc:1035] Waiting for connection context
0810 03:31:58.262965 (+ 136us) server_negotiation.cc:299] Negotiation successful
0810 03:31:58.263255 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260810 03:31:58.270164 30539 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5db20d63e80f485d820b6cb2659ee542: Bootstrap starting.
I20260810 03:31:58.274581 30658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:31:58.184820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55572 (local address 127.27.161.62:43211)
0810 03:31:58.184990 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0810 03:31:58.184995 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:31:58.214869 (+ 29874us) server_negotiation.cc:407] Connection header received
0810 03:31:58.215082 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:31:58.215087 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:31:58.215169 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:31:58.215280 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:31:58.230685 (+ 15405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:31:58.231838 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:31:58.243065 (+ 11227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:31:58.243776 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:31:58.243891 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:31:58.255115 (+ 11224us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:31:58.255124 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:31:58.255129 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:31:58.255201 (+ 72us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:31:58.259067 (+ 3866us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:31:58.269708 (+ 10641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:31:58.269715 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:31:58.269719 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:31:58.269941 (+ 222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:31:58.271242 (+ 1301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:31:58.271250 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:31:58.271254 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:31:58.271715 (+ 461us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:31:58.272023 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0810 03:31:58.273813 (+ 1790us) server_negotiation.cc:299] Negotiation successful
0810 03:31:58.274153 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260810 03:31:58.276458 30660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:31:58.184408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46632 (local address 127.27.161.60:40293)
0810 03:31:58.184610 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0810 03:31:58.184616 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:31:58.207648 (+ 23032us) server_negotiation.cc:407] Connection header received
0810 03:31:58.207915 (+ 267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:31:58.207921 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:31:58.208012 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:31:58.208116 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:31:58.238298 (+ 30182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:31:58.239326 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:31:58.244927 (+ 5601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:31:58.245603 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:31:58.245712 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:31:58.265077 (+ 19365us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:31:58.265083 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:31:58.265088 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:31:58.265129 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:31:58.268667 (+ 3538us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:31:58.270532 (+ 1865us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:31:58.270539 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:31:58.270542 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:31:58.270734 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:31:58.272529 (+ 1795us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:31:58.272536 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:31:58.272539 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:31:58.272926 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:31:58.273217 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0810 03:31:58.275839 (+ 2622us) server_negotiation.cc:299] Negotiation successful
0810 03:31:58.276105 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260810 03:31:58.277303 30539 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5db20d63e80f485d820b6cb2659ee542: Neither blocks nor log segments found. Creating new log.
I20260810 03:31:58.279664 30539 log.cc:824] T 00000000000000000000000000000000 P 5db20d63e80f485d820b6cb2659ee542: Log is configured to *not* fsync() on all Append() calls
I20260810 03:31:58.287729 30609 raft_consensus.cc:1261] T 00000000000000000000000000000000 P da7db9c2093848b9b767520fb1bf18f9 [term 1 FOLLOWER]: Refusing update from remote peer fa3f0f79c1034d2f8dc9edb8d61bcfc2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:31:58.288811 30662 consensus_queue.cc:1035] T 00000000000000000000000000000000 P fa3f0f79c1034d2f8dc9edb8d61bcfc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "da7db9c2093848b9b767520fb1bf18f9" member_type: VOTER last_known_addr { host: "127.27.161.60" port: 40293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:31:58.289917 30539 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5db20d63e80f485d820b6cb2659ee542: No bootstrap required, opened a new log
W20260810 03:31:58.300575 30578 tablet.cc:2369] T 00000000000000000000000000000000 P da7db9c2093848b9b767520fb1bf18f9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 03:31:58.302667 30539 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5db20d63e80f485d820b6cb2659ee542 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa3f0f79c1034d2f8dc9edb8d61bcfc2" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 43211 } } peers { permanent_uuid: "5db20d63e80f485d820b6cb2659ee542" member_type: VOTER last_known_addr { host: "127.27.161.61" port: 45273 } } peers { permanent_uuid: "da7db9c2093848b9b767520fb1bf18f9" member_type: VOTER last_known_addr { host: "127.27.161.60" port: 40293 } }
I20260810 03:31:58.303258 30539 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5db20d63e80f485d820b6cb2659ee542 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5db20d63e80f485d820b6cb2659ee542, State: Initialized, Role: FOLLOWER
I20260810 03:31:58.304054 30539 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5db20d63e80f485d820b6cb2659ee542 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa3f0f79c1034d2f8dc9edb8d61bcfc2" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 43211 } } peers { permanent_uuid: "5db20d63e80f485d820b6cb2659ee542" member_type: VOTER last_known_addr { host: "127.27.161.61" port: 45273 } } peers { permanent_uuid: "da7db9c2093848b9b767520fb1bf18f9" member_type: VOTER last_known_addr { host: "127.27.161.60" port: 40293 } }
I20260810 03:31:58.305610 30718 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5db20d63e80f485d820b6cb2659ee542 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa3f0f79c1034d2f8dc9edb8d61bcfc2" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 43211 } } peers { permanent_uuid: "5db20d63e80f485d820b6cb2659ee542" member_type: VOTER last_known_addr { host: "127.27.161.61" port: 45273 } } peers { permanent_uuid: "da7db9c2093848b9b767520fb1bf18f9" member_type: VOTER last_known_addr { host: "127.27.161.60" port: 40293 } } }
I20260810 03:31:58.306471 30718 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5db20d63e80f485d820b6cb2659ee542 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 03:31:58.306802 30539 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5db20d63e80f485d820b6cb2659ee542 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:31:58.307248 30496 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5db20d63e80f485d820b6cb2659ee542 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:31:58.307210 30671 sys_catalog.cc:455] T 00000000000000000000000000000000 P da7db9c2093848b9b767520fb1bf18f9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fa3f0f79c1034d2f8dc9edb8d61bcfc2. Latest consensus state: current_term: 1 leader_uuid: "fa3f0f79c1034d2f8dc9edb8d61bcfc2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa3f0f79c1034d2f8dc9edb8d61bcfc2" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 43211 } } peers { permanent_uuid: "5db20d63e80f485d820b6cb2659ee542" member_type: VOTER last_known_addr { host: "127.27.161.61" port: 45273 } } peers { permanent_uuid: "da7db9c2093848b9b767520fb1bf18f9" member_type: VOTER last_known_addr { host: "127.27.161.60" port: 40293 } } }
I20260810 03:31:58.307519 30671 sys_catalog.cc:458] T 00000000000000000000000000000000 P da7db9c2093848b9b767520fb1bf18f9 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 03:31:58.307698 30497 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fa3f0f79c1034d2f8dc9edb8d61bcfc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "5db20d63e80f485d820b6cb2659ee542"
I20260810 03:31:58.311626 30496 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5db20d63e80f485d820b6cb2659ee542 [term 1 FOLLOWER]: Refusing update from remote peer fa3f0f79c1034d2f8dc9edb8d61bcfc2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:31:58.313426 30714 mvcc.cc:205] Tried to move back new op lower bound from 7316818814084567040 to 7316818813666947072. Current Snapshot: MvccSnapshot[applied={T|T < 7316818814084567040}]
I20260810 03:31:58.315454 30497 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5db20d63e80f485d820b6cb2659ee542 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa3f0f79c1034d2f8dc9edb8d61bcfc2 in term 1.
I20260810 03:31:58.317870 30684 consensus_queue.cc:1035] T 00000000000000000000000000000000 P fa3f0f79c1034d2f8dc9edb8d61bcfc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5db20d63e80f485d820b6cb2659ee542" member_type: VOTER last_known_addr { host: "127.27.161.61" port: 45273 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:31:58.329583 30684 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa3f0f79c1034d2f8dc9edb8d61bcfc2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "fa3f0f79c1034d2f8dc9edb8d61bcfc2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa3f0f79c1034d2f8dc9edb8d61bcfc2" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 43211 } } peers { permanent_uuid: "5db20d63e80f485d820b6cb2659ee542" member_type: VOTER last_known_addr { host: "127.27.161.61" port: 45273 } } peers { permanent_uuid: "da7db9c2093848b9b767520fb1bf18f9" member_type: VOTER last_known_addr { host: "127.27.161.60" port: 40293 } } }
I20260810 03:31:58.329900 30684 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa3f0f79c1034d2f8dc9edb8d61bcfc2 [sys.catalog]: This master's current role is: LEADER
I20260810 03:31:58.331048 30707 catalog_manager.cc:1347] Generated new cluster ID: f2ff700bf2564e9e975cd8bcfcedebe8
I20260810 03:31:58.331162 30707 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 03:31:58.334203 30671 sys_catalog.cc:455] T 00000000000000000000000000000000 P da7db9c2093848b9b767520fb1bf18f9 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "fa3f0f79c1034d2f8dc9edb8d61bcfc2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa3f0f79c1034d2f8dc9edb8d61bcfc2" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 43211 } } peers { permanent_uuid: "5db20d63e80f485d820b6cb2659ee542" member_type: VOTER last_known_addr { host: "127.27.161.61" port: 45273 } } peers { permanent_uuid: "da7db9c2093848b9b767520fb1bf18f9" member_type: VOTER last_known_addr { host: "127.27.161.60" port: 40293 } } }
I20260810 03:31:58.334466 30671 sys_catalog.cc:458] T 00000000000000000000000000000000 P da7db9c2093848b9b767520fb1bf18f9 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 03:31:58.352299 30707 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 03:31:58.354439 30707 catalog_manager.cc:1504] Loading token signing keys...
I20260810 03:31:58.363019 30718 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5db20d63e80f485d820b6cb2659ee542 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fa3f0f79c1034d2f8dc9edb8d61bcfc2. Latest consensus state: current_term: 1 leader_uuid: "fa3f0f79c1034d2f8dc9edb8d61bcfc2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa3f0f79c1034d2f8dc9edb8d61bcfc2" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 43211 } } peers { permanent_uuid: "5db20d63e80f485d820b6cb2659ee542" member_type: VOTER last_known_addr { host: "127.27.161.61" port: 45273 } } peers { permanent_uuid: "da7db9c2093848b9b767520fb1bf18f9" member_type: VOTER last_known_addr { host: "127.27.161.60" port: 40293 } } }
I20260810 03:31:58.363485 30718 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5db20d63e80f485d820b6cb2659ee542 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 03:31:58.364004 30730 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5db20d63e80f485d820b6cb2659ee542 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "fa3f0f79c1034d2f8dc9edb8d61bcfc2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa3f0f79c1034d2f8dc9edb8d61bcfc2" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 43211 } } peers { permanent_uuid: "5db20d63e80f485d820b6cb2659ee542" member_type: VOTER last_known_addr { host: "127.27.161.61" port: 45273 } } peers { permanent_uuid: "da7db9c2093848b9b767520fb1bf18f9" member_type: VOTER last_known_addr { host: "127.27.161.60" port: 40293 } } }
I20260810 03:31:58.364436 30730 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5db20d63e80f485d820b6cb2659ee542 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 03:31:58.367509 30684 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa3f0f79c1034d2f8dc9edb8d61bcfc2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "fa3f0f79c1034d2f8dc9edb8d61bcfc2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa3f0f79c1034d2f8dc9edb8d61bcfc2" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 43211 } } peers { permanent_uuid: "5db20d63e80f485d820b6cb2659ee542" member_type: VOTER last_known_addr { host: "127.27.161.61" port: 45273 } } peers { permanent_uuid: "da7db9c2093848b9b767520fb1bf18f9" member_type: VOTER last_known_addr { host: "127.27.161.60" port: 40293 } } }
I20260810 03:31:58.367857 30684 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa3f0f79c1034d2f8dc9edb8d61bcfc2 [sys.catalog]: This master's current role is: LEADER
I20260810 03:31:58.375418 30707 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fa3f0f79c1034d2f8dc9edb8d61bcfc2: Generated new TSK 0
I20260810 03:31:58.376782 30707 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 03:31:58.409463 30747 catalog_manager.cc:1259] Loaded cluster ID: f2ff700bf2564e9e975cd8bcfcedebe8
I20260810 03:31:58.409668 30747 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 5db20d63e80f485d820b6cb2659ee542: loading cluster ID for follower catalog manager: success
I20260810 03:31:58.413772 30747 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 5db20d63e80f485d820b6cb2659ee542: acquiring CA information for follower catalog manager: success
I20260810 03:31:58.416952 30747 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 5db20d63e80f485d820b6cb2659ee542: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260810 03:31:58.496511 30658 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:31:58.482374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55586 (local address 127.27.161.62:43211)
0810 03:31:58.482612 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0810 03:31:58.482619 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:31:58.494041 (+ 11422us) server_negotiation.cc:407] Connection header received
0810 03:31:58.494642 (+ 601us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:31:58.494649 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:31:58.494745 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:31:58.494834 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:31:58.496239 (+ 1405us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55586: BlockingRecv error: recv got EOF from 127.0.0.1:55586 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
W20260810 03:31:58.499027 30660 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:31:58.483141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46656 (local address 127.27.161.60:40293)
0810 03:31:58.483364 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0810 03:31:58.483370 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:31:58.496005 (+ 12635us) server_negotiation.cc:407] Connection header received
0810 03:31:58.497256 (+ 1251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:31:58.497264 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:31:58.497357 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:31:58.497433 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:31:58.498754 (+ 1321us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46656: BlockingRecv error: recv got EOF from 127.0.0.1:46656 (error 108)
Metrics: {"server-negotiator.queue_time_us":95}
W20260810 03:31:58.499509 30546 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:31:58.496876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49792 (local address 127.27.161.61:45273)
0810 03:31:58.497289 (+ 413us) server_negotiation.cc:206] Beginning negotiation
0810 03:31:58.497295 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:31:58.497815 (+ 520us) server_negotiation.cc:407] Connection header received
0810 03:31:58.498107 (+ 292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:31:58.498124 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:31:58.498229 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:31:58.498301 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:31:58.499265 (+ 964us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49792: BlockingRecv error: recv got EOF from 127.0.0.1:49792 (error 108)
Metrics: {"server-negotiator.queue_time_us":254}
I20260810 03:31:58.545758 30546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:31:58.537406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49808 (local address 127.27.161.61:45273)
0810 03:31:58.537607 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0810 03:31:58.537614 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:31:58.538168 (+ 554us) server_negotiation.cc:407] Connection header received
0810 03:31:58.538436 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:31:58.538442 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:31:58.538663 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:31:58.538849 (+ 186us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 03:31:58.539341 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:31:58.540490 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:31:58.542670 (+ 2180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:31:58.543355 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:31:58.543488 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:31:58.544239 (+ 751us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 03:31:58.544958 (+ 719us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 03:31:58.545126 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0810 03:31:58.545304 (+ 178us) server_negotiation.cc:299] Negotiation successful
0810 03:31:58.545414 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
W20260810 03:31:58.556108 30660 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:31:58.553584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46668 (local address 127.27.161.60:40293)
0810 03:31:58.553765 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0810 03:31:58.553770 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:31:58.554506 (+ 736us) server_negotiation.cc:407] Connection header received
0810 03:31:58.554756 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:31:58.554761 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:31:58.554917 (+ 156us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:31:58.555028 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:31:58.555858 (+ 830us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46668: BlockingRecv error: recv got EOF from 127.0.0.1:46668 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
I20260810 03:31:58.572122 30546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:31:58.561068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49822 (local address 127.27.161.61:45273)
0810 03:31:58.561291 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0810 03:31:58.561297 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:31:58.565608 (+ 4311us) server_negotiation.cc:407] Connection header received
0810 03:31:58.565817 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:31:58.565823 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:31:58.565921 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:31:58.566050 (+ 129us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 03:31:58.566643 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:31:58.567828 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:31:58.569802 (+ 1974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:31:58.570543 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:31:58.570679 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:31:58.571154 (+ 475us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 03:31:58.571325 (+ 171us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 03:31:58.571432 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0810 03:31:58.571690 (+ 258us) server_negotiation.cc:299] Negotiation successful
0810 03:31:58.571784 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
W20260810 03:31:58.581454 30660 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:31:58.579824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46682 (local address 127.27.161.60:40293)
0810 03:31:58.580031 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0810 03:31:58.580037 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:31:58.580058 (+ 21us) server_negotiation.cc:407] Connection header received
0810 03:31:58.580145 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:31:58.580150 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:31:58.580240 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:31:58.580348 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:31:58.581207 (+ 859us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46682: BlockingRecv error: recv got EOF from 127.0.0.1:46682 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
I20260810 03:31:58.596011 30546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:31:58.587454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49838 (local address 127.27.161.61:45273)
0810 03:31:58.587636 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0810 03:31:58.587642 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:31:58.588440 (+ 798us) server_negotiation.cc:407] Connection header received
0810 03:31:58.588594 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:31:58.588600 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:31:58.588767 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:31:58.588916 (+ 149us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 03:31:58.590414 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:31:58.591637 (+ 1223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:31:58.593751 (+ 2114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:31:58.594487 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:31:58.594653 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:31:58.595137 (+ 484us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 03:31:58.595298 (+ 161us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 03:31:58.595394 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0810 03:31:58.595549 (+ 155us) server_negotiation.cc:299] Negotiation successful
0810 03:31:58.595653 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
W20260810 03:31:58.605837 30660 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:31:58.603822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46684 (local address 127.27.161.60:40293)
0810 03:31:58.604009 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0810 03:31:58.604015 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:31:58.604038 (+ 23us) server_negotiation.cc:407] Connection header received
0810 03:31:58.604265 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:31:58.604271 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:31:58.604372 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:31:58.604452 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:31:58.605572 (+ 1120us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46684: BlockingRecv error: recv got EOF from 127.0.0.1:46684 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
I20260810 03:31:58.621841 30546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:31:58.614432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49844 (local address 127.27.161.61:45273)
0810 03:31:58.614693 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0810 03:31:58.614700 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:31:58.615225 (+ 525us) server_negotiation.cc:407] Connection header received
0810 03:31:58.615436 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:31:58.615442 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:31:58.615576 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:31:58.615687 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 03:31:58.616272 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:31:58.617487 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:31:58.619541 (+ 2054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:31:58.620302 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:31:58.620463 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:31:58.620970 (+ 507us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 03:31:58.621113 (+ 143us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 03:31:58.621224 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0810 03:31:58.621440 (+ 216us) server_negotiation.cc:299] Negotiation successful
0810 03:31:58.621538 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120}
W20260810 03:31:58.639030 30660 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:31:58.632420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46686 (local address 127.27.161.60:40293)
0810 03:31:58.632719 (+ 299us) server_negotiation.cc:206] Beginning negotiation
0810 03:31:58.632726 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:31:58.637082 (+ 4356us) server_negotiation.cc:407] Connection header received
0810 03:31:58.637315 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:31:58.637322 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:31:58.637437 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:31:58.637576 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:31:58.638782 (+ 1206us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46686: BlockingRecv error: recv got EOF from 127.0.0.1:46686 (error 108)
Metrics: {"server-negotiator.queue_time_us":160}
I20260810 03:31:58.663546 30546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:31:58.656250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49832 (local address 127.27.161.61:45273)
0810 03:31:58.656433 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0810 03:31:58.656439 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:31:58.656990 (+ 551us) server_negotiation.cc:407] Connection header received
0810 03:31:58.657215 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:31:58.657221 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:31:58.657334 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:31:58.657450 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 03:31:58.657940 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:31:58.659195 (+ 1255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:31:58.661237 (+ 2042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:31:58.661943 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:31:58.662141 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:31:58.662628 (+ 487us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 03:31:58.662833 (+ 205us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 03:31:58.662947 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0810 03:31:58.663105 (+ 158us) server_negotiation.cc:299] Negotiation successful
0810 03:31:58.663200 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
W20260810 03:31:58.684731 30660 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:31:58.679120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36426 (local address 127.27.161.60:40293)
0810 03:31:58.679310 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0810 03:31:58.679317 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:31:58.682653 (+ 3336us) server_negotiation.cc:407] Connection header received
0810 03:31:58.683298 (+ 645us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:31:58.683305 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:31:58.683417 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:31:58.683543 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:31:58.684465 (+ 922us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36426: BlockingRecv error: recv got EOF from 127.0.0.1:36426 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
I20260810 03:31:58.712212 30546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:31:58.704708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49846 (local address 127.27.161.61:45273)
0810 03:31:58.704911 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0810 03:31:58.704918 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:31:58.705481 (+ 563us) server_negotiation.cc:407] Connection header received
0810 03:31:58.705713 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:31:58.705720 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:31:58.705833 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:31:58.705991 (+ 158us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 03:31:58.706642 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:31:58.707901 (+ 1259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:31:58.709859 (+ 1958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:31:58.710609 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:31:58.710763 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:31:58.711312 (+ 549us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 03:31:58.711513 (+ 201us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 03:31:58.711614 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0810 03:31:58.711780 (+ 166us) server_negotiation.cc:299] Negotiation successful
0810 03:31:58.711886 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
W20260810 03:31:58.730325 30660 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:31:58.727828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36430 (local address 127.27.161.60:40293)
0810 03:31:58.728035 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0810 03:31:58.728041 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:31:58.728618 (+ 577us) server_negotiation.cc:407] Connection header received
0810 03:31:58.728862 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:31:58.728869 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:31:58.728983 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:31:58.729108 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:31:58.730078 (+ 970us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36430: BlockingRecv error: recv got EOF from 127.0.0.1:36430 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
I20260810 03:31:58.774988 30546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:31:58.766753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49854 (local address 127.27.161.61:45273)
0810 03:31:58.766986 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0810 03:31:58.766993 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:31:58.767514 (+ 521us) server_negotiation.cc:407] Connection header received
0810 03:31:58.767740 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:31:58.767747 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:31:58.767841 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:31:58.767947 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 03:31:58.768384 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:31:58.769530 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:31:58.771791 (+ 2261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:31:58.772489 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:31:58.772659 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:31:58.773392 (+ 733us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 03:31:58.773557 (+ 165us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 03:31:58.773830 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0810 03:31:58.774085 (+ 255us) server_negotiation.cc:299] Negotiation successful
0810 03:31:58.774646 (+ 561us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
W20260810 03:31:58.783623 30660 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:31:58.781763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36446 (local address 127.27.161.60:40293)
0810 03:31:58.782014 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0810 03:31:58.782020 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:31:58.782271 (+ 251us) server_negotiation.cc:407] Connection header received
0810 03:31:58.782428 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:31:58.782434 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:31:58.782550 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:31:58.782670 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:31:58.783379 (+ 709us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36446: BlockingRecv error: recv got EOF from 127.0.0.1:36446 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
I20260810 03:31:58.856768 30546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:31:58.850907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49870 (local address 127.27.161.61:45273)
0810 03:31:58.851155 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0810 03:31:58.851162 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:31:58.851528 (+ 366us) server_negotiation.cc:407] Connection header received
0810 03:31:58.851696 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:31:58.851700 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:31:58.851776 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:31:58.851867 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 03:31:58.852317 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:31:58.853089 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:31:58.855071 (+ 1982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:31:58.855574 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:31:58.855676 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:31:58.856142 (+ 466us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 03:31:58.856258 (+ 116us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 03:31:58.856323 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0810 03:31:58.856456 (+ 133us) server_negotiation.cc:299] Negotiation successful
0810 03:31:58.856527 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
W20260810 03:31:58.863973 30660 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:31:58.862125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36452 (local address 127.27.161.60:40293)
0810 03:31:58.862290 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0810 03:31:58.862294 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:31:58.862658 (+ 364us) server_negotiation.cc:407] Connection header received
0810 03:31:58.862829 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:31:58.862833 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:31:58.862905 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:31:58.862979 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:31:58.863729 (+ 750us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36452: BlockingRecv error: recv got EOF from 127.0.0.1:36452 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
I20260810 03:31:59.000775 30546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:31:58.995844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49872 (local address 127.27.161.61:45273)
0810 03:31:58.996063 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0810 03:31:58.996070 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:31:58.996402 (+ 332us) server_negotiation.cc:407] Connection header received
0810 03:31:58.996665 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:31:58.996669 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:31:58.996747 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:31:58.996822 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 03:31:58.997276 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:31:58.998021 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:31:58.999291 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:31:58.999680 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:31:58.999769 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:31:59.000126 (+ 357us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 03:31:59.000229 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 03:31:59.000288 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0810 03:31:59.000426 (+ 138us) server_negotiation.cc:299] Negotiation successful
0810 03:31:59.000492 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
W20260810 03:31:59.008145 30660 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:31:59.006267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36464 (local address 127.27.161.60:40293)
0810 03:31:59.006475 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0810 03:31:59.006481 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:31:59.006797 (+ 316us) server_negotiation.cc:407] Connection header received
0810 03:31:59.007048 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:31:59.007053 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:31:59.007128 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:31:59.007212 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:31:59.007911 (+ 699us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36464: BlockingRecv error: recv got EOF from 127.0.0.1:36464 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
I20260810 03:31:59.242324 30713 catalog_manager.cc:1259] Loaded cluster ID: f2ff700bf2564e9e975cd8bcfcedebe8
I20260810 03:31:59.242462 30713 catalog_manager.cc:1552] T 00000000000000000000000000000000 P da7db9c2093848b9b767520fb1bf18f9: loading cluster ID for follower catalog manager: success
I20260810 03:31:59.245572 30713 catalog_manager.cc:1574] T 00000000000000000000000000000000 P da7db9c2093848b9b767520fb1bf18f9: acquiring CA information for follower catalog manager: success
I20260810 03:31:59.247789 30713 catalog_manager.cc:1602] T 00000000000000000000000000000000 P da7db9c2093848b9b767520fb1bf18f9: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 03:31:59.277048 30546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:31:59.271732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49874 (local address 127.27.161.61:45273)
0810 03:31:59.271909 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0810 03:31:59.271915 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:31:59.272339 (+ 424us) server_negotiation.cc:407] Connection header received
0810 03:31:59.272545 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:31:59.272549 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:31:59.272637 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:31:59.272727 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 03:31:59.273217 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:31:59.273998 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:31:59.275350 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:31:59.275790 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:31:59.275904 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:31:59.276341 (+ 437us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 03:31:59.276455 (+ 114us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 03:31:59.276531 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0810 03:31:59.276693 (+ 162us) server_negotiation.cc:299] Negotiation successful
0810 03:31:59.276780 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260810 03:31:59.291088 30660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:31:59.283841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36478 (local address 127.27.161.60:40293)
0810 03:31:59.284088 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0810 03:31:59.284094 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:31:59.284415 (+ 321us) server_negotiation.cc:407] Connection header received
0810 03:31:59.284626 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:31:59.284631 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:31:59.284797 (+ 166us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:31:59.284907 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 03:31:59.285450 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:31:59.286398 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:31:59.288447 (+ 2049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:31:59.288916 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:31:59.289049 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:31:59.289733 (+ 684us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 03:31:59.290389 (+ 656us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 03:31:59.290513 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0810 03:31:59.290640 (+ 127us) server_negotiation.cc:299] Negotiation successful
0810 03:31:59.290734 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105}
I20260810 03:31:59.316442 30546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:31:59.308457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49876 (local address 127.27.161.61:45273)
0810 03:31:59.308642 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0810 03:31:59.308648 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:31:59.310471 (+ 1823us) server_negotiation.cc:407] Connection header received
0810 03:31:59.310676 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:31:59.310682 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:31:59.310775 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:31:59.310867 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 03:31:59.311512 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:31:59.312587 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:31:59.314196 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:31:59.314815 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:31:59.314958 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:31:59.315585 (+ 627us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 03:31:59.315728 (+ 143us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 03:31:59.315794 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0810 03:31:59.316016 (+ 222us) server_negotiation.cc:299] Negotiation successful
0810 03:31:59.316112 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260810 03:31:59.316732 30917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:31:59.299950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54108 (local address 127.27.161.62:43211)
0810 03:31:59.307650 (+ 7700us) server_negotiation.cc:206] Beginning negotiation
0810 03:31:59.307659 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 03:31:59.308298 (+ 639us) server_negotiation.cc:407] Connection header received
0810 03:31:59.309120 (+ 822us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:31:59.309127 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:31:59.309366 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:31:59.309562 (+ 196us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 03:31:59.310730 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:31:59.311728 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:31:59.314197 (+ 2469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:31:59.314835 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:31:59.314975 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:31:59.315564 (+ 589us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 03:31:59.316050 (+ 486us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 03:31:59.316279 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0810 03:31:59.316342 (+ 63us) server_negotiation.cc:299] Negotiation successful
0810 03:31:59.316434 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7529,"thread_start_us":7390,"threads_started":1}
I20260810 03:31:59.319628 30660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:31:59.300002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36482 (local address 127.27.161.60:40293)
0810 03:31:59.300250 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0810 03:31:59.300257 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:31:59.309219 (+ 8962us) server_negotiation.cc:407] Connection header received
0810 03:31:59.309509 (+ 290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:31:59.309516 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:31:59.309624 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:31:59.309821 (+ 197us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 03:31:59.312801 (+ 2980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:31:59.314004 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:31:59.316899 (+ 2895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:31:59.317592 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:31:59.317698 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:31:59.318811 (+ 1113us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 03:31:59.318963 (+ 152us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 03:31:59.319025 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0810 03:31:59.319269 (+ 244us) server_negotiation.cc:299] Negotiation successful
0810 03:31:59.319368 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260810 03:31:59.322822 28292 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iBH5/build/asan/bin/kudu pid 30330
I20260810 03:31:59.324025 30422 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:31:59.437796 28292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iBH5/build/asan/bin/kudu with pid 30330
W20260810 03:31:59.475278 30572 connection.cc:535] server connection from 127.0.0.1:46606 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260810 03:31:59.475729 28292 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iBH5/build/asan/bin/kudu pid 30443
I20260810 03:31:59.476966 30533 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:31:59.585055 28292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iBH5/build/asan/bin/kudu with pid 30443
I20260810 03:31:59.611356 28292 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iBH5/build/asan/bin/kudu pid 30550
I20260810 03:31:59.612455 30639 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:31:59.727768 28292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iBH5/build/asan/bin/kudu with pid 30550
2026-08-10T03:31:59Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (4883 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786332336319544-28292-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 03:31:59 dist-test-slave-vpvm krb5kdc[30932](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 03:31:59 dist-test-slave-vpvm krb5kdc[30932](info): set up 2 sockets
Aug 10 03:31:59 dist-test-slave-vpvm krb5kdc[30932](info): commencing operation
krb5kdc: starting...
W20260810 03:32:01.006891 28292 mini_kdc.cc:121] Time spent starting KDC: real 1.227s user 0.000s sys 0.062s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786332336319544-28292-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786332336319544-28292-0/krb5kdc/test-user.keytab.
Aug 10 03:32:01 dist-test-slave-vpvm krb5kdc[30932](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332721, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T03:32:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T03:32:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.161.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.161.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.161.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.62.keytab.
Entry for principal kudu/127.27.161.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.62.keytab.
I20260810 03:32:01.375020 28292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iBH5/build/asan/bin/kudu
/tmp/dist-test-task24iBH5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786332336319544-28292-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786332336319544-28292-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786332336319544-28292-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786332336319544-28292-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.161.62:41793
--webserver_interface=127.27.161.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.62.keytab
--principal=kudu/127.27.161.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.161.20:40151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.161.62:41793,127.27.161.61:42455,127.27.161.60:35303
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786332336319544-28292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786332336319544-28292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786332336319544-28292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:32:01.746496 30958 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260810 03:32:01.746868 30958 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:32:01.746973 30958 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:32:01.747081 30958 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:32:01.754606 30958 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:32:01.754719 30958 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:32:01.754784 30958 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:32:01.754859 30958 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:32:01.754935 30958 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:32:01.765355 30958 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.161.20:40151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786332336319544-28292-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786332336319544-28292-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.161.62:41793,127.27.161.61:42455,127.27.161.60:35303
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.62.keytab
--principal=kudu/127.27.161.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.161.62:41793
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786332336319544-28292-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.161.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786332336319544-28292-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:32:01.766819 30958 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:32:01.768239 30958 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:32:01.780550 30965 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:32:01.780529 30966 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:32:01.780486 30969 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:32:01.781366 30958 server_base.cc:1034] running on GCE node
I20260810 03:32:01.782255 30958 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:32:01.783794 30958 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:32:01.785043 30958 hybrid_clock.cc:648] HybridClock initialized: now 1786332721784962 us; error 53 us; skew 500 ppm
I20260810 03:32:01.785395 30958 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:32:01 dist-test-slave-vpvm krb5kdc[30932](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332721, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.161.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:32:01.789076 30958 init.cc:377] Logged in from keytab as kudu/127.27.161.62@KRBTEST.COM (short username kudu)
I20260810 03:32:01.790848 30958 webserver.cc:460] Webserver started at http://127.27.161.62:42241/ using document root <none> and password file <none>
I20260810 03:32:01.791731 30958 fs_manager.cc:359] Metadata directory not provided
I20260810 03:32:01.791888 30958 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:32:01.792260 30958 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:32:01.795538 30958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786332336319544-28292-0/minicluster-data/master-0/data/instance:
uuid: "c0c9f82ffab643f9865b6ef08d2191fb"
format_stamp: "Formatted at 2026-08-10 03:32:01 on dist-test-slave-vpvm"
server_key: "e51e129b28fd78a733dc8b77a936d860"
server_key_iv: "d0f69b7de8230e80593c8242f1a68e99"
server_key_version: "encryptionkey@0"
I20260810 03:32:01.796380 30958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786332336319544-28292-0/minicluster-data/master-0/wal/instance:
uuid: "c0c9f82ffab643f9865b6ef08d2191fb"
format_stamp: "Formatted at 2026-08-10 03:32:01 on dist-test-slave-vpvm"
server_key: "e51e129b28fd78a733dc8b77a936d860"
server_key_iv: "d0f69b7de8230e80593c8242f1a68e99"
server_key_version: "encryptionkey@0"
I20260810 03:32:01.802336 30958 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260810 03:32:01.806455 30980 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:32:01.807987 30958 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260810 03:32:01.808194 30958 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786332336319544-28292-0/minicluster-data/master-0/data,/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786332336319544-28292-0/minicluster-data/master-0/wal
uuid: "c0c9f82ffab643f9865b6ef08d2191fb"
format_stamp: "Formatted at 2026-08-10 03:32:01 on dist-test-slave-vpvm"
server_key: "e51e129b28fd78a733dc8b77a936d860"
server_key_iv: "d0f69b7de8230e80593c8242f1a68e99"
server_key_version: "encryptionkey@0"
I20260810 03:32:01.808426 30958 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786332336319544-28292-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786332336319544-28292-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786332336319544-28292-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:32:01.825151 30958 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:32:01.829509 30958 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:32:01.829803 30958 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:32:01.848443 30958 rpc_server.cc:307] RPC server started. Bound to: 127.27.161.62:41793
I20260810 03:32:01.848462 31053 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.161.62:41793 every 8 connection(s)
I20260810 03:32:01.849852 30958 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786332336319544-28292-0/minicluster-data/master-0/data/info.pb
I20260810 03:32:01.852474 28292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iBH5/build/asan/bin/kudu as pid 30958
I20260810 03:32:01.852666 28292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786332336319544-28292-0/minicluster-data/master-0/wal/instance
I20260810 03:32:01.853086 28292 external_mini_cluster.cc:1428] Setting key cf3438b102d7528d19f6a15d831cf24a
I20260810 03:32:01.853749 31054 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:32:01.860299 31054 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.161.62" port: 41793 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 03:32:01 dist-test-slave-vpvm krb5kdc[30932](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332721, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.161.62@KRBTEST.COM for kudu/127.27.161.62@KRBTEST.COM
I20260810 03:32:01.884698 31057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:32:01.863562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53680 (local address 127.27.161.62:41793)
0810 03:32:01.865172 (+ 1610us) server_negotiation.cc:206] Beginning negotiation
0810 03:32:01.865180 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 03:32:01.865232 (+ 52us) server_negotiation.cc:407] Connection header received
0810 03:32:01.866943 (+ 1711us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:32:01.866951 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:32:01.867261 (+ 310us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:32:01.867453 (+ 192us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:32:01.869020 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:32:01.870272 (+ 1252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:32:01.872253 (+ 1981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:32:01.872849 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:32:01.872990 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:32:01.877075 (+ 4085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:32:01.877086 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:32:01.877103 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:32:01.877138 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:32:01.879990 (+ 2852us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:32:01.880943 (+ 953us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:32:01.880952 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:32:01.880954 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:32:01.881131 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:32:01.881603 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:32:01.881608 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:32:01.881610 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:32:01.882245 (+ 635us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:32:01.882482 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0810 03:32:01.883185 (+ 703us) server_negotiation.cc:299] Negotiation successful
0810 03:32:01.883557 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":45,"server-negotiator.queue_time_us":1399,"spinlock_wait_cycles":54016,"thread_start_us":1192,"threads_started":1}
I20260810 03:32:01.884725 31056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:32:01.863567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:41793 (local address 127.0.0.1:53680)
0810 03:32:01.864517 (+ 950us) negotiation.cc:107] Waiting for socket to connect
0810 03:32:01.864595 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0810 03:32:01.866545 (+ 1950us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:32:01.867920 (+ 1375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:32:01.867926 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:32:01.868471 (+ 545us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:32:01.868813 (+ 342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:32:01.868830 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:32:01.870452 (+ 1622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:32:01.870458 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:32:01.872094 (+ 1636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:32:01.872109 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:32:01.873119 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:32:01.873126 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:32:01.873294 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:32:01.874058 (+ 764us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:32:01.874096 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:32:01.876806 (+ 2710us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:32:01.880153 (+ 3347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:32:01.880166 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:32:01.880179 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:32:01.880785 (+ 606us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:32:01.881253 (+ 468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:32:01.881258 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:32:01.881261 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:32:01.881461 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:32:01.882395 (+ 934us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:32:01.882403 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:32:01.882752 (+ 349us) client_negotiation.cc:769] Sending connection context
0810 03:32:01.883042 (+ 290us) client_negotiation.cc:240] Negotiation successful
0810 03:32:01.883754 (+ 712us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":629,"mutex_wait_us":128,"thread_start_us":375,"threads_started":1}
I20260810 03:32:01.887974 31054 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.161.61" port: 42455 } has no permanent_uuid. Determining permanent_uuid...
I20260810 03:32:01.889245 31056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:32:01.888723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.61:42455 (local address 127.0.0.1:50252)
0810 03:32:01.888933 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0810 03:32:01.889085 (+ 152us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.61:42455: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
W20260810 03:32:01.889590 30983 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.161.61:42455: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.27.161.61@KRBTEST.COM; defaulting to no policy
I20260810 03:32:01.891211 31056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:32:01.890813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.61:42455 (local address 127.0.0.1:50268)
0810 03:32:01.890962 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0810 03:32:01.891043 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.61:42455: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260810 03:32:01.891429 31054 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.161.61:42455: Network error: Client connection negotiation failed: client connection to 127.27.161.61:42455: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.161.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.161.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.61.keytab.
Entry for principal kudu/127.27.161.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.61.keytab.
I20260810 03:32:01.935602 28292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iBH5/build/asan/bin/kudu
/tmp/dist-test-task24iBH5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786332336319544-28292-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786332336319544-28292-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786332336319544-28292-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786332336319544-28292-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.161.61:42455
--webserver_interface=127.27.161.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.61.keytab
--principal=kudu/127.27.161.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.161.20:40151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.161.62:41793,127.27.161.61:42455,127.27.161.60:35303
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786332336319544-28292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786332336319544-28292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786332336319544-28292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 03:32:01.940829 31054 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.161.61" port: 42455 } attempt: 1
I20260810 03:32:01.942210 31056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:32:01.941610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.61:42455 (local address 127.0.0.1:50284)
0810 03:32:01.941855 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0810 03:32:01.942030 (+ 175us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.61:42455: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260810 03:32:01.943470 31056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:32:01.942998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.61:42455 (local address 127.0.0.1:50288)
0810 03:32:01.943191 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0810 03:32:01.943298 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.61:42455: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
W20260810 03:32:01.943723 31054 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.161.61:42455: Network error: Client connection negotiation failed: client connection to 127.27.161.61:42455: connect: Connection refused (error 111)
I20260810 03:32:02.012027 31054 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.161.61" port: 42455 } attempt: 2
I20260810 03:32:02.013454 31056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:32:02.012945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.61:42455 (local address 127.0.0.1:50294)
0810 03:32:02.013213 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0810 03:32:02.013305 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.61:42455: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260810 03:32:02.014518 31056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:32:02.014124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.61:42455 (local address 127.0.0.1:50308)
0810 03:32:02.014272 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0810 03:32:02.014375 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.61:42455: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260810 03:32:02.014741 31054 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.161.61:42455: Network error: Client connection negotiation failed: client connection to 127.27.161.61:42455: connect: Connection refused (error 111)
I20260810 03:32:02.106081 31054 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.161.61" port: 42455 } attempt: 3
I20260810 03:32:02.107388 31056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:32:02.106898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.61:42455 (local address 127.0.0.1:50310)
0810 03:32:02.107141 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0810 03:32:02.107240 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.61:42455: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
I20260810 03:32:02.108578 31056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:32:02.108156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.61:42455 (local address 127.0.0.1:50312)
0810 03:32:02.108307 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0810 03:32:02.108388 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.61:42455: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260810 03:32:02.108799 31054 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.161.61:42455: Network error: Client connection negotiation failed: client connection to 127.27.161.61:42455: connect: Connection refused (error 111)
I20260810 03:32:02.252032 31054 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.161.61" port: 42455 } attempt: 4
I20260810 03:32:02.253940 31056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:32:02.253113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.61:42455 (local address 127.0.0.1:50318)
0810 03:32:02.253639 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0810 03:32:02.253756 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.61:42455: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144}
I20260810 03:32:02.255378 31056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:32:02.254840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.61:42455 (local address 127.0.0.1:50330)
0810 03:32:02.255110 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0810 03:32:02.255200 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.61:42455: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137}
W20260810 03:32:02.255780 31054 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.161.61:42455: Network error: Client connection negotiation failed: client connection to 127.27.161.61:42455: connect: Connection refused (error 111)
W20260810 03:32:02.303308 31062 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260810 03:32:02.303694 31062 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:32:02.303793 31062 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:32:02.303910 31062 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:32:02.311436 31062 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:32:02.311568 31062 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:32:02.311663 31062 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:32:02.311735 31062 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:32:02.311821 31062 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:32:02.322818 31062 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.161.20:40151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786332336319544-28292-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786332336319544-28292-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.27.161.62:41793,127.27.161.61:42455,127.27.161.60:35303
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.61.keytab
--principal=kudu/127.27.161.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.161.61:42455
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786332336319544-28292-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.27.161.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786332336319544-28292-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:32:02.324303 31062 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:32:02.325711 31062 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:32:02.338778 31069 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:32:02.338796 31071 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:32:02.338826 31074 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:32:02.339769 31062 server_base.cc:1034] running on GCE node
I20260810 03:32:02.340672 31062 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:32:02.342343 31062 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:32:02.343703 31062 hybrid_clock.cc:648] HybridClock initialized: now 1786332722343617 us; error 49 us; skew 500 ppm
I20260810 03:32:02.344094 31062 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:32:02 dist-test-slave-vpvm krb5kdc[30932](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332722, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.161.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:32:02.347553 31062 init.cc:377] Logged in from keytab as kudu/127.27.161.61@KRBTEST.COM (short username kudu)
I20260810 03:32:02.349244 31062 webserver.cc:460] Webserver started at http://127.27.161.61:43539/ using document root <none> and password file <none>
I20260810 03:32:02.350060 31062 fs_manager.cc:359] Metadata directory not provided
I20260810 03:32:02.350165 31062 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:32:02.350456 31062 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:32:02.353710 31062 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786332336319544-28292-0/minicluster-data/master-1/data/instance:
uuid: "46b6aa835c274b3fbdca0b6d2ff09ac7"
format_stamp: "Formatted at 2026-08-10 03:32:02 on dist-test-slave-vpvm"
server_key: "f992f894be3d4fa8d9d90e8a6ada26de"
server_key_iv: "eff76dde87a8c47d26ab4fc4188c7100"
server_key_version: "encryptionkey@0"
I20260810 03:32:02.354624 31062 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786332336319544-28292-0/minicluster-data/master-1/wal/instance:
uuid: "46b6aa835c274b3fbdca0b6d2ff09ac7"
format_stamp: "Formatted at 2026-08-10 03:32:02 on dist-test-slave-vpvm"
server_key: "f992f894be3d4fa8d9d90e8a6ada26de"
server_key_iv: "eff76dde87a8c47d26ab4fc4188c7100"
server_key_version: "encryptionkey@0"
I20260810 03:32:02.360698 31062 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.003s
I20260810 03:32:02.364804 31088 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:32:02.366458 31062 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260810 03:32:02.366671 31062 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786332336319544-28292-0/minicluster-data/master-1/data,/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786332336319544-28292-0/minicluster-data/master-1/wal
uuid: "46b6aa835c274b3fbdca0b6d2ff09ac7"
format_stamp: "Formatted at 2026-08-10 03:32:02 on dist-test-slave-vpvm"
server_key: "f992f894be3d4fa8d9d90e8a6ada26de"
server_key_iv: "eff76dde87a8c47d26ab4fc4188c7100"
server_key_version: "encryptionkey@0"
I20260810 03:32:02.366910 31062 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786332336319544-28292-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786332336319544-28292-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786332336319544-28292-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:32:02.396747 31062 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:32:02.401261 31062 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:32:02.401556 31062 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:32:02.420264 31062 rpc_server.cc:307] RPC server started. Bound to: 127.27.161.61:42455
I20260810 03:32:02.420290 31165 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.161.61:42455 every 8 connection(s)
I20260810 03:32:02.421715 31062 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786332336319544-28292-0/minicluster-data/master-1/data/info.pb
I20260810 03:32:02.424850 28292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iBH5/build/asan/bin/kudu as pid 31062
I20260810 03:32:02.425060 28292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786332336319544-28292-0/minicluster-data/master-1/wal/instance
I20260810 03:32:02.425626 28292 external_mini_cluster.cc:1428] Setting key d3b8d2be94176582f3f324a040f00cf4
I20260810 03:32:02.426221 31166 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:32:02.435735 31166 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.161.62" port: 41793 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 03:32:02 dist-test-slave-vpvm krb5kdc[30932](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332722, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.161.61@KRBTEST.COM for kudu/127.27.161.62@KRBTEST.COM
I20260810 03:32:02.461422 31168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:32:02.439324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53682 (local address 127.27.161.62:41793)
0810 03:32:02.439925 (+ 601us) server_negotiation.cc:206] Beginning negotiation
0810 03:32:02.439931 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:32:02.441009 (+ 1078us) server_negotiation.cc:407] Connection header received
0810 03:32:02.443310 (+ 2301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:32:02.443317 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:32:02.443411 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:32:02.443539 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:32:02.445986 (+ 2447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:32:02.446736 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:32:02.449027 (+ 2291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:32:02.449454 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:32:02.449558 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:32:02.454605 (+ 5047us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:32:02.454612 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:32:02.454616 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:32:02.454683 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:32:02.457447 (+ 2764us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:32:02.458653 (+ 1206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:32:02.458658 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:32:02.458661 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:32:02.458812 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:32:02.459359 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:32:02.459364 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:32:02.459366 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:32:02.459718 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:32:02.459848 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0810 03:32:02.460897 (+ 1049us) server_negotiation.cc:299] Negotiation successful
0810 03:32:02.461112 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":465,"thread_start_us":302,"threads_started":1}
I20260810 03:32:02.462922 31169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:32:02.439963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:41793 (local address 127.0.0.1:53682)
0810 03:32:02.440813 (+ 850us) negotiation.cc:107] Waiting for socket to connect
0810 03:32:02.440862 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0810 03:32:02.442977 (+ 2115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:32:02.443922 (+ 945us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:32:02.443935 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:32:02.444797 (+ 862us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:32:02.445671 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:32:02.445701 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:32:02.446925 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:32:02.446931 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:32:02.448844 (+ 1913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:32:02.448864 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:32:02.449724 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:32:02.449731 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:32:02.450066 (+ 335us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:32:02.450849 (+ 783us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:32:02.450902 (+ 53us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:32:02.454277 (+ 3375us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:32:02.457662 (+ 3385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:32:02.457673 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:32:02.457693 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:32:02.458481 (+ 788us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:32:02.458942 (+ 461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:32:02.458947 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:32:02.458950 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:32:02.459203 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:32:02.459870 (+ 667us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:32:02.459879 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:32:02.460395 (+ 516us) client_negotiation.cc:769] Sending connection context
0810 03:32:02.460748 (+ 353us) client_negotiation.cc:240] Negotiation successful
0810 03:32:02.461348 (+ 600us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":562,"thread_start_us":340,"threads_started":1}
I20260810 03:32:02.464830 31166 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.161.61" port: 42455 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.27.161.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.161.60@KRBTEST.COM" created.
Aug 10 03:32:02 dist-test-slave-vpvm krb5kdc[30932](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332722, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.161.61@KRBTEST.COM for kudu/127.27.161.61@KRBTEST.COM
I20260810 03:32:02.480432 31169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:32:02.465632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.61:42455 (local address 127.0.0.1:50346)
0810 03:32:02.465830 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0810 03:32:02.465853 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0810 03:32:02.466001 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:32:02.467056 (+ 1055us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:32:02.467061 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:32:02.467473 (+ 412us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:32:02.467656 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:32:02.467666 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:32:02.469189 (+ 1523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:32:02.469194 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:32:02.470116 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:32:02.470129 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:32:02.471008 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:32:02.471014 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:32:02.471193 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:32:02.471807 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:32:02.471825 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:32:02.474320 (+ 2495us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:32:02.477887 (+ 3567us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:32:02.477892 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:32:02.477895 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:32:02.478412 (+ 517us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:32:02.478797 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:32:02.478801 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:32:02.478804 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:32:02.478965 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:32:02.479640 (+ 675us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:32:02.479644 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:32:02.479752 (+ 108us) client_negotiation.cc:769] Sending connection context
0810 03:32:02.479846 (+ 94us) client_negotiation.cc:240] Negotiation successful
0810 03:32:02.480107 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260810 03:32:02.481706 31173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:32:02.465867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50346 (local address 127.27.161.61:42455)
0810 03:32:02.466536 (+ 669us) server_negotiation.cc:206] Beginning negotiation
0810 03:32:02.466541 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:32:02.466567 (+ 26us) server_negotiation.cc:407] Connection header received
0810 03:32:02.466635 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:32:02.466641 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:32:02.466828 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:32:02.466946 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:32:02.467858 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:32:02.469047 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:32:02.470261 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:32:02.470856 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:32:02.470974 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:32:02.474550 (+ 3576us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:32:02.474564 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:32:02.474583 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:32:02.474619 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:32:02.477736 (+ 3117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:32:02.478550 (+ 814us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:32:02.478559 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:32:02.478562 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:32:02.478696 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:32:02.479077 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:32:02.479081 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:32:02.479083 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:32:02.479523 (+ 440us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:32:02.480936 (+ 1413us) server_negotiation.cc:1035] Waiting for connection context
0810 03:32:02.481121 (+ 185us) server_negotiation.cc:299] Negotiation successful
0810 03:32:02.481350 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":525,"thread_start_us":437,"threads_started":1}
I20260810 03:32:02.487838 31166 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.161.60" port: 35303 } has no permanent_uuid. Determining permanent_uuid...
I20260810 03:32:02.489136 31169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:32:02.488551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.60:35303 (local address 127.0.0.1:47344)
0810 03:32:02.488771 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0810 03:32:02.488946 (+ 175us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.60:35303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260810 03:32:02.489413 31098 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.161.60:35303: connect: Connection refused (error 111)
I20260810 03:32:02.491025 31169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:32:02.490628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.60:35303 (local address 127.0.0.1:47350)
0810 03:32:02.490790 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0810 03:32:02.490887 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.60:35303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260810 03:32:02.491258 31166 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.161.60:35303: Network error: Client connection negotiation failed: client connection to 127.27.161.60:35303: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.161.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.60.keytab.
Entry for principal kudu/127.27.161.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.60.keytab.
I20260810 03:32:02.514127 28292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task24iBH5/build/asan/bin/kudu
/tmp/dist-test-task24iBH5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786332336319544-28292-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786332336319544-28292-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786332336319544-28292-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786332336319544-28292-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.161.60:35303
--webserver_interface=127.27.161.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.60.keytab
--principal=kudu/127.27.161.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.161.20:40151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.161.62:41793,127.27.161.61:42455,127.27.161.60:35303
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786332336319544-28292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786332336319544-28292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786332336319544-28292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 03:32:02.540704 31166 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.161.60" port: 35303 } attempt: 1
I20260810 03:32:02.542140 31169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:32:02.541538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.60:35303 (local address 127.0.0.1:47366)
0810 03:32:02.541776 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0810 03:32:02.541912 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.60:35303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260810 03:32:02.543336 31169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:32:02.542862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.60:35303 (local address 127.0.0.1:47370)
0810 03:32:02.543061 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0810 03:32:02.543165 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.60:35303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260810 03:32:02.543560 31166 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.161.60:35303: Network error: Client connection negotiation failed: client connection to 127.27.161.60:35303: connect: Connection refused (error 111)
I20260810 03:32:02.555043 31054 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.161.61" port: 42455 } attempt: 5
Aug 10 03:32:02 dist-test-slave-vpvm krb5kdc[30932](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332721, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.161.62@KRBTEST.COM for kudu/127.27.161.61@KRBTEST.COM
I20260810 03:32:02.569191 31173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:32:02.555899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50348 (local address 127.27.161.61:42455)
0810 03:32:02.556079 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0810 03:32:02.556083 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:32:02.556103 (+ 20us) server_negotiation.cc:407] Connection header received
0810 03:32:02.556246 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:32:02.556249 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:32:02.556337 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:32:02.556420 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:32:02.557348 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:32:02.558104 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:32:02.559763 (+ 1659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:32:02.560213 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:32:02.560322 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:32:02.563958 (+ 3636us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:32:02.563964 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:32:02.563967 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:32:02.564004 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:32:02.566563 (+ 2559us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:32:02.567237 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:32:02.567241 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:32:02.567244 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:32:02.567373 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:32:02.567764 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:32:02.567767 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:32:02.567769 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:32:02.568083 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:32:02.568219 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0810 03:32:02.568616 (+ 397us) server_negotiation.cc:299] Negotiation successful
0810 03:32:02.568837 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260810 03:32:02.569190 31056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:32:02.555763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.61:42455 (local address 127.0.0.1:50348)
0810 03:32:02.555978 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0810 03:32:02.555999 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0810 03:32:02.556132 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:32:02.556465 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:32:02.556470 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:32:02.556977 (+ 507us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:32:02.557213 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:32:02.557226 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:32:02.558230 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:32:02.558236 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:32:02.559572 (+ 1336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:32:02.559592 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:32:02.560376 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:32:02.560382 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:32:02.560614 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:32:02.561225 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:32:02.561248 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:32:02.563737 (+ 2489us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:32:02.566712 (+ 2975us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:32:02.566717 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:32:02.566722 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:32:02.567106 (+ 384us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:32:02.567490 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:32:02.567495 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:32:02.567499 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:32:02.567652 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:32:02.568203 (+ 551us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:32:02.568208 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:32:02.568363 (+ 155us) client_negotiation.cc:769] Sending connection context
0810 03:32:02.568488 (+ 125us) client_negotiation.cc:240] Negotiation successful
0810 03:32:02.568829 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67}
I20260810 03:32:02.570349 31054 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.161.60" port: 35303 } has no permanent_uuid. Determining permanent_uuid...
I20260810 03:32:02.571471 31056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:32:02.571040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.60:35303 (local address 127.0.0.1:47384)
0810 03:32:02.571215 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0810 03:32:02.571316 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.60:35303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260810 03:32:02.572548 31056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:32:02.572157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.60:35303 (local address 127.0.0.1:47388)
0810 03:32:02.572326 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0810 03:32:02.572422 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.60:35303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
W20260810 03:32:02.572767 31054 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.161.60:35303: Network error: Client connection negotiation failed: client connection to 127.27.161.60:35303: connect: Connection refused (error 111)
I20260810 03:32:02.611847 31166 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.161.60" port: 35303 } attempt: 2
I20260810 03:32:02.613176 31169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:32:02.612596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.60:35303 (local address 127.0.0.1:47402)
0810 03:32:02.612837 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0810 03:32:02.612955 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.60:35303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260810 03:32:02.614446 31169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:32:02.614024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.60:35303 (local address 127.0.0.1:47414)
0810 03:32:02.614193 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0810 03:32:02.614288 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.60:35303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260810 03:32:02.614679 31166 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.161.60:35303: Network error: Client connection negotiation failed: client connection to 127.27.161.60:35303: connect: Connection refused (error 111)
I20260810 03:32:02.624006 31054 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.161.60" port: 35303 } attempt: 1
I20260810 03:32:02.625300 31056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:32:02.624790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.60:35303 (local address 127.0.0.1:47416)
0810 03:32:02.625024 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0810 03:32:02.625120 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.60:35303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260810 03:32:02.626386 31056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:32:02.626015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.60:35303 (local address 127.0.0.1:47432)
0810 03:32:02.626177 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0810 03:32:02.626261 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.60:35303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260810 03:32:02.626605 31054 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.161.60:35303: Network error: Client connection negotiation failed: client connection to 127.27.161.60:35303: connect: Connection refused (error 111)
I20260810 03:32:02.694816 31054 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.161.60" port: 35303 } attempt: 2
I20260810 03:32:02.696148 31056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:32:02.695563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.60:35303 (local address 127.0.0.1:47448)
0810 03:32:02.695833 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0810 03:32:02.695959 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.60:35303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260810 03:32:02.697141 31056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:32:02.696828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.60:35303 (local address 127.0.0.1:47450)
0810 03:32:02.696950 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0810 03:32:02.697022 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.60:35303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260810 03:32:02.697359 31054 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.161.60:35303: Network error: Client connection negotiation failed: client connection to 127.27.161.60:35303: connect: Connection refused (error 111)
I20260810 03:32:02.705926 31166 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.161.60" port: 35303 } attempt: 3
I20260810 03:32:02.707211 31169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:32:02.706647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.60:35303 (local address 127.0.0.1:47452)
0810 03:32:02.706885 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0810 03:32:02.706996 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.60:35303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260810 03:32:02.708380 31169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:32:02.707859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.60:35303 (local address 127.0.0.1:47466)
0810 03:32:02.708119 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0810 03:32:02.708220 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.60:35303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142}
W20260810 03:32:02.708578 31166 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.161.60:35303: Network error: Client connection negotiation failed: client connection to 127.27.161.60:35303: connect: Connection refused (error 111)
I20260810 03:32:02.803606 31054 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.161.60" port: 35303 } attempt: 3
I20260810 03:32:02.804728 31056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:32:02.804256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.60:35303 (local address 127.0.0.1:47468)
0810 03:32:02.804455 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0810 03:32:02.804551 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.60:35303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260810 03:32:02.805980 31056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:32:02.805507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.60:35303 (local address 127.0.0.1:47478)
0810 03:32:02.805751 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0810 03:32:02.805833 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.60:35303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129}
W20260810 03:32:02.806245 31054 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.161.60:35303: Network error: Client connection negotiation failed: client connection to 127.27.161.60:35303: connect: Connection refused (error 111)
I20260810 03:32:02.851823 31166 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.161.60" port: 35303 } attempt: 4
I20260810 03:32:02.853201 31169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:32:02.852611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.60:35303 (local address 127.0.0.1:47488)
0810 03:32:02.852888 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0810 03:32:02.852999 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.60:35303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260810 03:32:02.854460 31169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:32:02.854037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.60:35303 (local address 127.0.0.1:47504)
0810 03:32:02.854223 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0810 03:32:02.854321 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.161.60:35303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260810 03:32:02.854759 31166 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.161.60:35303: Network error: Client connection negotiation failed: client connection to 127.27.161.60:35303: connect: Connection refused (error 111)
W20260810 03:32:02.881441 31178 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260810 03:32:02.881822 31178 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:32:02.881917 31178 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:32:02.882061 31178 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:32:02.889493 31178 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:32:02.889617 31178 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:32:02.889704 31178 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:32:02.889772 31178 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:32:02.889860 31178 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:32:02.900195 31178 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.161.20:40151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786332336319544-28292-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786332336319544-28292-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.27.161.62:41793,127.27.161.61:42455,127.27.161.60:35303
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786332336319544-28292-0/krb5kdc/kudu_127.27.161.60.keytab
--principal=kudu/127.27.161.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.161.60:35303
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786332336319544-28292-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.27.161.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786332336319544-28292-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:32:02.901585 31178 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:32:02.903003 31178 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:32:02.915689 31188 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:32:02.915766 31190 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:32:02.915710 31184 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:32:02.916589 31178 server_base.cc:1034] running on GCE node
I20260810 03:32:02.917392 31178 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:32:02.918959 31178 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:32:02.920186 31178 hybrid_clock.cc:648] HybridClock initialized: now 1786332722920114 us; error 42 us; skew 500 ppm
I20260810 03:32:02.920540 31178 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:32:02 dist-test-slave-vpvm krb5kdc[30932](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332722, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.161.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:32:02.923859 31178 init.cc:377] Logged in from keytab as kudu/127.27.161.60@KRBTEST.COM (short username kudu)
I20260810 03:32:02.925452 31178 webserver.cc:460] Webserver started at http://127.27.161.60:35487/ using document root <none> and password file <none>
I20260810 03:32:02.926250 31178 fs_manager.cc:359] Metadata directory not provided
I20260810 03:32:02.926359 31178 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:32:02.926695 31178 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:32:02.929806 31178 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786332336319544-28292-0/minicluster-data/master-2/data/instance:
uuid: "97352ed50d564be5aeca141f1cb3218e"
format_stamp: "Formatted at 2026-08-10 03:32:02 on dist-test-slave-vpvm"
server_key: "08294dc03b254801c6643d9e5f751140"
server_key_iv: "b264cef9ae1c359be0105b25afba9224"
server_key_version: "encryptionkey@0"
I20260810 03:32:02.930692 31178 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786332336319544-28292-0/minicluster-data/master-2/wal/instance:
uuid: "97352ed50d564be5aeca141f1cb3218e"
format_stamp: "Formatted at 2026-08-10 03:32:02 on dist-test-slave-vpvm"
server_key: "08294dc03b254801c6643d9e5f751140"
server_key_iv: "b264cef9ae1c359be0105b25afba9224"
server_key_version: "encryptionkey@0"
I20260810 03:32:02.935988 31178 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260810 03:32:02.939574 31200 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:32:02.940945 31178 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260810 03:32:02.941118 31178 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786332336319544-28292-0/minicluster-data/master-2/data,/tmp/dist-test-task24iBH5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786332336319544-28292-0/minicluster-data/master-2/wal
uuid: "97352ed50d564be5aeca141f1cb3218e"
format_stamp: "Formatted at 2026-08-10 03:32:02 on dist-test-slave-vpvm"
server_key: "08294dc03b254801c6643d9e5f751140"
server_key_iv: "b264cef9ae1c359be0105b25afba9224"
server_key_version: "encryptionkey@0"
I20260810 03:32:02.941334 31178 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786332336319544-28292-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786332336319544-28292-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786332336319544-28292-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:32:02.956230 31178 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:32:02.960881 31178 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:32:02.961198 31178 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:32:02.978794 31178 rpc_server.cc:307] RPC server started. Bound to: 127.27.161.60:35303
I20260810 03:32:02.978814 31277 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.161.60:35303 every 8 connection(s)
I20260810 03:32:02.980312 31178 server_base.cc:1166] Dumped server information to /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786332336319544-28292-0/minicluster-data/master-2/data/info.pb
I20260810 03:32:02.981648 28292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task24iBH5/build/asan/bin/kudu as pid 31178
I20260810 03:32:02.981829 28292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task24iBH5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786332336319544-28292-0/minicluster-data/master-2/wal/instance
I20260810 03:32:02.982364 28292 external_mini_cluster.cc:1428] Setting key 220367ea110f622bec4e17b4755f3b6a
I20260810 03:32:02.983453 31054 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.161.60" port: 35303 } attempt: 4
I20260810 03:32:02.985306 31278 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 10 03:32:02 dist-test-slave-vpvm krb5kdc[30932](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332721, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.161.62@KRBTEST.COM for kudu/127.27.161.60@KRBTEST.COM
I20260810 03:32:02.995114 31278 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.161.62" port: 41793 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 03:32:02 dist-test-slave-vpvm krb5kdc[30932](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332721, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.161.60@KRBTEST.COM
I20260810 03:32:03.006716 31056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:32:02.984116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.60:35303 (local address 127.0.0.1:47520)
0810 03:32:02.984397 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0810 03:32:02.984417 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 03:32:02.984637 (+ 220us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:32:02.988142 (+ 3505us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:32:02.988149 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:32:02.988674 (+ 525us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:32:02.988867 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:32:02.988879 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:32:02.990944 (+ 2065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:32:02.990951 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:32:02.992184 (+ 1233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:32:02.992199 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:32:02.993212 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:32:02.993218 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:32:02.993441 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:32:02.994153 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:32:02.994180 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:32:02.996700 (+ 2520us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:32:03.003071 (+ 6371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:32:03.003077 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:32:03.003081 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:32:03.003670 (+ 589us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:32:03.004241 (+ 571us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:32:03.004245 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:32:03.004247 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:32:03.004493 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:32:03.005526 (+ 1033us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:32:03.005532 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:32:03.005720 (+ 188us) client_negotiation.cc:769] Sending connection context
0810 03:32:03.005891 (+ 171us) client_negotiation.cc:240] Negotiation successful
0810 03:32:03.006270 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167}
I20260810 03:32:03.007720 31282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:32:02.985197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47520 (local address 127.27.161.60:35303)
0810 03:32:02.985794 (+ 597us) server_negotiation.cc:206] Beginning negotiation
0810 03:32:02.985798 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:32:02.985817 (+ 19us) server_negotiation.cc:407] Connection header received
0810 03:32:02.987258 (+ 1441us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:32:02.987281 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:32:02.987750 (+ 469us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:32:02.988160 (+ 410us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:32:02.989061 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:32:02.990716 (+ 1655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:32:02.992354 (+ 1638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:32:02.993047 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:32:02.993220 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:32:02.997052 (+ 3832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:32:02.997064 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:32:02.997080 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:32:02.997125 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:32:03.002894 (+ 5769us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:32:03.003821 (+ 927us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:32:03.003831 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:32:03.003841 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:32:03.004096 (+ 255us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:32:03.004624 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:32:03.004629 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:32:03.004632 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:32:03.005362 (+ 730us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:32:03.005541 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0810 03:32:03.006165 (+ 624us) server_negotiation.cc:299] Negotiation successful
0810 03:32:03.006517 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":29,"server-negotiator.queue_time_us":494,"spinlock_wait_cycles":768,"thread_start_us":305,"threads_started":1}
I20260810 03:32:03.007896 31281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:32:02.984305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47506 (local address 127.27.161.60:35303)
0810 03:32:02.985335 (+ 1030us) server_negotiation.cc:206] Beginning negotiation
0810 03:32:02.985352 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0810 03:32:02.985491 (+ 139us) server_negotiation.cc:407] Connection header received
0810 03:32:02.987284 (+ 1793us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:32:02.987294 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:32:02.987754 (+ 460us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:32:02.988146 (+ 392us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:32:02.990305 (+ 2159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:32:02.991670 (+ 1365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:32:02.993533 (+ 1863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:32:02.994153 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:32:02.994278 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:32:02.999361 (+ 5083us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:32:02.999370 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:32:02.999375 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:32:02.999412 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:32:03.003818 (+ 4406us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:32:03.004653 (+ 835us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:32:03.004657 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:32:03.004659 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:32:03.004845 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:32:03.005351 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:32:03.005356 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:32:03.005358 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:32:03.005737 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:32:03.007265 (+ 1528us) server_negotiation.cc:1035] Waiting for connection context
0810 03:32:03.007369 (+ 104us) server_negotiation.cc:299] Negotiation successful
0810 03:32:03.007555 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":766,"spinlock_wait_cycles":77568,"thread_start_us":428,"threads_started":1}
Aug 10 03:32:03 dist-test-slave-vpvm krb5kdc[30932](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332722, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.161.60@KRBTEST.COM for kudu/127.27.161.62@KRBTEST.COM
I20260810 03:32:03.015302 31054 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c0c9f82ffab643f9865b6ef08d2191fb: Bootstrap starting.
I20260810 03:32:03.018335 31285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:32:02.998643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.62:41793 (local address 127.0.0.1:53694)
0810 03:32:02.999530 (+ 887us) negotiation.cc:107] Waiting for socket to connect
0810 03:32:02.999593 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0810 03:32:02.999764 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:32:03.000253 (+ 489us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:32:03.000262 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:32:03.000987 (+ 725us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:32:03.001361 (+ 374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:32:03.001372 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:32:03.002583 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:32:03.002587 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:32:03.004176 (+ 1589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:32:03.004192 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:32:03.005201 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:32:03.005207 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:32:03.005398 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:32:03.008238 (+ 2840us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:32:03.008268 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:32:03.010949 (+ 2681us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:32:03.015179 (+ 4230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:32:03.015194 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:32:03.015202 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:32:03.015793 (+ 591us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:32:03.016203 (+ 410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:32:03.016209 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:32:03.016212 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:32:03.016391 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:32:03.017097 (+ 706us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:32:03.017105 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:32:03.017323 (+ 218us) client_negotiation.cc:769] Sending connection context
0810 03:32:03.017536 (+ 213us) client_negotiation.cc:240] Negotiation successful
0810 03:32:03.017878 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":607,"thread_start_us":456,"threads_started":1}
I20260810 03:32:03.018358 31284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:32:02.998525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53694 (local address 127.27.161.62:41793)
0810 03:32:02.999097 (+ 572us) server_negotiation.cc:206] Beginning negotiation
0810 03:32:02.999103 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:32:02.999895 (+ 792us) server_negotiation.cc:407] Connection header received
0810 03:32:02.999984 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:32:02.999990 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:32:03.000089 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:32:03.000194 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:32:03.001516 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:32:03.002394 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:32:03.004354 (+ 1960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:32:03.004988 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:32:03.005136 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:32:03.011183 (+ 6047us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:32:03.011190 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:32:03.011195 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:32:03.011237 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:32:03.015025 (+ 3788us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:32:03.015901 (+ 876us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:32:03.015906 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:32:03.015910 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:32:03.016077 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:32:03.016510 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:32:03.016515 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:32:03.016518 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:32:03.016926 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:32:03.017144 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0810 03:32:03.017654 (+ 510us) server_negotiation.cc:299] Negotiation successful
0810 03:32:03.017884 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":419,"thread_start_us":271,"threads_started":1}
I20260810 03:32:03.019567 31054 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c0c9f82ffab643f9865b6ef08d2191fb: Neither blocks nor log segments found. Creating new log.
I20260810 03:32:03.020380 31278 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.161.61" port: 42455 } has no permanent_uuid. Determining permanent_uuid...
I20260810 03:32:03.020871 31054 log.cc:824] T 00000000000000000000000000000000 P c0c9f82ffab643f9865b6ef08d2191fb: Log is configured to *not* fsync() on all Append() calls
I20260810 03:32:03.024199 31054 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c0c9f82ffab643f9865b6ef08d2191fb: No bootstrap required, opened a new log
Aug 10 03:32:03 dist-test-slave-vpvm krb5kdc[30932](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332721, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.161.62@KRBTEST.COM
Aug 10 03:32:03 dist-test-slave-vpvm krb5kdc[30932](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332722, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.161.60@KRBTEST.COM for kudu/127.27.161.61@KRBTEST.COM
I20260810 03:32:03.032780 31288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:32:03.012018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53710 (local address 127.27.161.62:41793)
0810 03:32:03.015350 (+ 3332us) server_negotiation.cc:206] Beginning negotiation
0810 03:32:03.015354 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:32:03.015372 (+ 18us) server_negotiation.cc:407] Connection header received
0810 03:32:03.015410 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:32:03.015413 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:32:03.015479 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:32:03.015580 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:32:03.017186 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:32:03.018412 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:32:03.019980 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:32:03.020598 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:32:03.020729 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:32:03.026241 (+ 5512us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:32:03.026250 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:32:03.026256 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:32:03.026303 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:32:03.029645 (+ 3342us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:32:03.030556 (+ 911us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:32:03.030560 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:32:03.030562 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:32:03.030736 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:32:03.031122 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:32:03.031125 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:32:03.031127 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:32:03.031521 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:32:03.031812 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0810 03:32:03.032134 (+ 322us) server_negotiation.cc:299] Negotiation successful
0810 03:32:03.032367 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3219,"thread_start_us":1511,"threads_started":1}
I20260810 03:32:03.034854 31054 raft_consensus.cc:348] T 00000000000000000000000000000000 P c0c9f82ffab643f9865b6ef08d2191fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0c9f82ffab643f9865b6ef08d2191fb" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 41793 } } peers { permanent_uuid: "46b6aa835c274b3fbdca0b6d2ff09ac7" member_type: VOTER last_known_addr { host: "127.27.161.61" port: 42455 } } peers { permanent_uuid: "97352ed50d564be5aeca141f1cb3218e" member_type: VOTER last_known_addr { host: "127.27.161.60" port: 35303 } }
I20260810 03:32:03.035462 31054 raft_consensus.cc:729] T 00000000000000000000000000000000 P c0c9f82ffab643f9865b6ef08d2191fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0c9f82ffab643f9865b6ef08d2191fb, State: Initialized, Role: FOLLOWER
I20260810 03:32:03.036108 31054 consensus_queue.cc:260] T 00000000000000000000000000000000 P c0c9f82ffab643f9865b6ef08d2191fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0c9f82ffab643f9865b6ef08d2191fb" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 41793 } } peers { permanent_uuid: "46b6aa835c274b3fbdca0b6d2ff09ac7" member_type: VOTER last_known_addr { host: "127.27.161.61" port: 42455 } } peers { permanent_uuid: "97352ed50d564be5aeca141f1cb3218e" member_type: VOTER last_known_addr { host: "127.27.161.60" port: 35303 } }
I20260810 03:32:03.037230 31292 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0c9f82ffab643f9865b6ef08d2191fb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0c9f82ffab643f9865b6ef08d2191fb" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 41793 } } peers { permanent_uuid: "46b6aa835c274b3fbdca0b6d2ff09ac7" member_type: VOTER last_known_addr { host: "127.27.161.61" port: 42455 } } peers { permanent_uuid: "97352ed50d564be5aeca141f1cb3218e" member_type: VOTER last_known_addr { host: "127.27.161.60" port: 35303 } } }
I20260810 03:32:03.037791 31292 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0c9f82ffab643f9865b6ef08d2191fb [sys.catalog]: This master's current role is: FOLLOWER
I20260810 03:32:03.038183 31173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:32:03.021344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50352 (local address 127.27.161.61:42455)
0810 03:32:03.021530 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0810 03:32:03.021534 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:32:03.022260 (+ 726us) server_negotiation.cc:407] Connection header received
0810 03:32:03.022540 (+ 280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:32:03.022546 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:32:03.022660 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:32:03.022777 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:32:03.023902 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:32:03.024724 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:32:03.025945 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:32:03.026553 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:32:03.026687 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:32:03.030571 (+ 3884us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:32:03.030579 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:32:03.030584 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:32:03.030631 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:32:03.034581 (+ 3950us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:32:03.035504 (+ 923us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:32:03.035511 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:32:03.035514 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:32:03.035783 (+ 269us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:32:03.036325 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:32:03.036331 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:32:03.036334 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:32:03.036739 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:32:03.036886 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0810 03:32:03.037464 (+ 578us) server_negotiation.cc:299] Negotiation successful
0810 03:32:03.037722 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260810 03:32:03.038834 31054 sys_catalog.cc:564] T 00000000000000000000000000000000 P c0c9f82ffab643f9865b6ef08d2191fb [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:32:03.039162 31285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:32:03.021196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.61:42455 (local address 127.0.0.1:50352)
0810 03:32:03.022159 (+ 963us) negotiation.cc:107] Waiting for socket to connect
0810 03:32:03.022180 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0810 03:32:03.022371 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:32:03.022973 (+ 602us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:32:03.022978 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:32:03.023474 (+ 496us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:32:03.023710 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:32:03.023724 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:32:03.024893 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:32:03.024896 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:32:03.025795 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:32:03.025808 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:32:03.026735 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:32:03.026741 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:32:03.026932 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:32:03.027493 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:32:03.027517 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:32:03.030330 (+ 2813us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:32:03.034738 (+ 4408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:32:03.034744 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:32:03.034748 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:32:03.035331 (+ 583us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:32:03.035970 (+ 639us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:32:03.035974 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:32:03.035978 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:32:03.036175 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:32:03.036996 (+ 821us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:32:03.037000 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:32:03.037183 (+ 183us) client_negotiation.cc:769] Sending connection context
0810 03:32:03.038449 (+ 1266us) client_negotiation.cc:240] Negotiation successful
0810 03:32:03.038796 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":813}
I20260810 03:32:03.040305 31278 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.161.60" port: 35303 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 03:32:03 dist-test-slave-vpvm krb5kdc[30932](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332721, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.161.61@KRBTEST.COM
Aug 10 03:32:03 dist-test-slave-vpvm krb5kdc[30932](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332722, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.161.60@KRBTEST.COM for kudu/127.27.161.60@KRBTEST.COM
I20260810 03:32:03.054519 31291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:32:03.034972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50360 (local address 127.27.161.61:42455)
0810 03:32:03.035885 (+ 913us) server_negotiation.cc:206] Beginning negotiation
0810 03:32:03.035889 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:32:03.035909 (+ 20us) server_negotiation.cc:407] Connection header received
0810 03:32:03.035950 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:32:03.035953 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:32:03.036007 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:32:03.036101 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:32:03.037794 (+ 1693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:32:03.038831 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:32:03.040323 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:32:03.040944 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:32:03.041058 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:32:03.047300 (+ 6242us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:32:03.047309 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:32:03.047314 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:32:03.047360 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:32:03.050820 (+ 3460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:32:03.051823 (+ 1003us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:32:03.051828 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:32:03.051831 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:32:03.052024 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:32:03.052535 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:32:03.052541 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:32:03.052545 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:32:03.052937 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:32:03.053206 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0810 03:32:03.053801 (+ 595us) server_negotiation.cc:299] Negotiation successful
0810 03:32:03.054093 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":807,"thread_start_us":592,"threads_started":1}
I20260810 03:32:03.057602 31285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:32:03.041073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.60:35303 (local address 127.0.0.1:47534)
0810 03:32:03.041262 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0810 03:32:03.041282 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 03:32:03.041546 (+ 264us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:32:03.042055 (+ 509us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:32:03.042060 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:32:03.042549 (+ 489us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:32:03.042765 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:32:03.042778 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:32:03.044176 (+ 1398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:32:03.044180 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:32:03.046166 (+ 1986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:32:03.046190 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:32:03.047304 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:32:03.047309 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:32:03.047533 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:32:03.048122 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:32:03.048148 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:32:03.050937 (+ 2789us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:32:03.054902 (+ 3965us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:32:03.054907 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:32:03.054911 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:32:03.055403 (+ 492us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:32:03.055779 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:32:03.055785 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:32:03.055789 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:32:03.055919 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:32:03.056614 (+ 695us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:32:03.056617 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:32:03.056729 (+ 112us) client_negotiation.cc:769] Sending connection context
0810 03:32:03.056847 (+ 118us) client_negotiation.cc:240] Negotiation successful
0810 03:32:03.057173 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":70}
I20260810 03:32:03.057632 31281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:32:03.041263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47534 (local address 127.27.161.60:35303)
0810 03:32:03.041433 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0810 03:32:03.041439 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:32:03.041457 (+ 18us) server_negotiation.cc:407] Connection header received
0810 03:32:03.041679 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:32:03.041683 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:32:03.041753 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:32:03.041833 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:32:03.042923 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:32:03.044004 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:32:03.046348 (+ 2344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:32:03.046976 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:32:03.047069 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:32:03.051128 (+ 4059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:32:03.051134 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:32:03.051138 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:32:03.051177 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:32:03.054745 (+ 3568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:32:03.055531 (+ 786us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:32:03.055536 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:32:03.055540 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:32:03.055669 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:32:03.056036 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:32:03.056040 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:32:03.056044 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:32:03.056435 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:32:03.056562 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0810 03:32:03.056983 (+ 421us) server_negotiation.cc:299] Negotiation successful
0810 03:32:03.057334 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":115,"server-negotiator.queue_time_us":59}
W20260810 03:32:03.061299 31308 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c0c9f82ffab643f9865b6ef08d2191fb: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 03:32:03.061482 31308 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 03:32:03.062991 31278 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 97352ed50d564be5aeca141f1cb3218e: Bootstrap starting.
I20260810 03:32:03.052433 30958 master_runner.cc:186] Error getting master registration for 127.27.161.61:42455: OK, instance_id { permanent_uuid: "46b6aa835c274b3fbdca0b6d2ff09ac7" instance_seqno: 1786332722415020 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260810 03:32:03.065908 30958 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 03:32:03.066888 28292 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 03:32:03.067413 31278 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 97352ed50d564be5aeca141f1cb3218e: Neither blocks nor log segments found. Creating new log.
I20260810 03:32:03.068527 31278 log.cc:824] T 00000000000000000000000000000000 P 97352ed50d564be5aeca141f1cb3218e: Log is configured to *not* fsync() on all Append() calls
I20260810 03:32:03.070386 31019 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "c0c9f82ffab643f9865b6ef08d2191fb"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53710
I20260810 03:32:03.070883 31019 raft_consensus.cc:482] T 00000000000000000000000000000000 P c0c9f82ffab643f9865b6ef08d2191fb [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260810 03:32:03.071094 31019 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c0c9f82ffab643f9865b6ef08d2191fb [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:32:03.075045 31019 raft_consensus.cc:504] T 00000000000000000000000000000000 P c0c9f82ffab643f9865b6ef08d2191fb [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0c9f82ffab643f9865b6ef08d2191fb" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 41793 } } peers { permanent_uuid: "46b6aa835c274b3fbdca0b6d2ff09ac7" member_type: VOTER last_known_addr { host: "127.27.161.61" port: 42455 } } peers { permanent_uuid: "97352ed50d564be5aeca141f1cb3218e" member_type: VOTER last_known_addr { host: "127.27.161.60" port: 35303 } }
I20260810 03:32:03.077412 31278 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 97352ed50d564be5aeca141f1cb3218e: No bootstrap required, opened a new log
I20260810 03:32:03.077440 31019 leader_election.cc:288] T 00000000000000000000000000000000 P c0c9f82ffab643f9865b6ef08d2191fb [CANDIDATE]: Term 1 election: Requested vote from peers 46b6aa835c274b3fbdca0b6d2ff09ac7 (127.27.161.61:42455), 97352ed50d564be5aeca141f1cb3218e (127.27.161.60:35303)
W20260810 03:32:03.082846 30990 tablet.cc:2369] T 00000000000000000000000000000000 P c0c9f82ffab643f9865b6ef08d2191fb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 03:32:03.089234 31278 raft_consensus.cc:348] T 00000000000000000000000000000000 P 97352ed50d564be5aeca141f1cb3218e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0c9f82ffab643f9865b6ef08d2191fb" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 41793 } } peers { permanent_uuid: "46b6aa835c274b3fbdca0b6d2ff09ac7" member_type: VOTER last_known_addr { host: "127.27.161.61" port: 42455 } } peers { permanent_uuid: "97352ed50d564be5aeca141f1cb3218e" member_type: VOTER last_known_addr { host: "127.27.161.60" port: 35303 } }
I20260810 03:32:03.089775 31278 raft_consensus.cc:729] T 00000000000000000000000000000000 P 97352ed50d564be5aeca141f1cb3218e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97352ed50d564be5aeca141f1cb3218e, State: Initialized, Role: FOLLOWER
I20260810 03:32:03.090380 31278 consensus_queue.cc:260] T 00000000000000000000000000000000 P 97352ed50d564be5aeca141f1cb3218e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0c9f82ffab643f9865b6ef08d2191fb" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 41793 } } peers { permanent_uuid: "46b6aa835c274b3fbdca0b6d2ff09ac7" member_type: VOTER last_known_addr { host: "127.27.161.61" port: 42455 } } peers { permanent_uuid: "97352ed50d564be5aeca141f1cb3218e" member_type: VOTER last_known_addr { host: "127.27.161.60" port: 35303 } }
I20260810 03:32:03.091902 31317 sys_catalog.cc:455] T 00000000000000000000000000000000 P 97352ed50d564be5aeca141f1cb3218e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0c9f82ffab643f9865b6ef08d2191fb" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 41793 } } peers { permanent_uuid: "46b6aa835c274b3fbdca0b6d2ff09ac7" member_type: VOTER last_known_addr { host: "127.27.161.61" port: 42455 } } peers { permanent_uuid: "97352ed50d564be5aeca141f1cb3218e" member_type: VOTER last_known_addr { host: "127.27.161.60" port: 35303 } } }
I20260810 03:32:03.092880 31317 sys_catalog.cc:458] T 00000000000000000000000000000000 P 97352ed50d564be5aeca141f1cb3218e [sys.catalog]: This master's current role is: FOLLOWER
I20260810 03:32:03.093170 31278 sys_catalog.cc:564] T 00000000000000000000000000000000 P 97352ed50d564be5aeca141f1cb3218e [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:32:03.093214 31242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c0c9f82ffab643f9865b6ef08d2191fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "97352ed50d564be5aeca141f1cb3218e"
I20260810 03:32:03.093688 31242 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 97352ed50d564be5aeca141f1cb3218e [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:32:03.097535 31242 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 97352ed50d564be5aeca141f1cb3218e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0c9f82ffab643f9865b6ef08d2191fb in term 1.
I20260810 03:32:03.098568 30988 leader_election.cc:302] T 00000000000000000000000000000000 P c0c9f82ffab643f9865b6ef08d2191fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 97352ed50d564be5aeca141f1cb3218e, c0c9f82ffab643f9865b6ef08d2191fb; no voters:
I20260810 03:32:03.099156 31292 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c0c9f82ffab643f9865b6ef08d2191fb [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:32:03.112895 31292 raft_consensus.cc:686] T 00000000000000000000000000000000 P c0c9f82ffab643f9865b6ef08d2191fb [term 1 LEADER]: Becoming Leader. State: Replica: c0c9f82ffab643f9865b6ef08d2191fb, State: Running, Role: LEADER
I20260810 03:32:03.113754 31292 consensus_queue.cc:237] T 00000000000000000000000000000000 P c0c9f82ffab643f9865b6ef08d2191fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0c9f82ffab643f9865b6ef08d2191fb" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 41793 } } peers { permanent_uuid: "46b6aa835c274b3fbdca0b6d2ff09ac7" member_type: VOTER last_known_addr { host: "127.27.161.61" port: 42455 } } peers { permanent_uuid: "97352ed50d564be5aeca141f1cb3218e" member_type: VOTER last_known_addr { host: "127.27.161.60" port: 35303 } }
I20260810 03:32:03.118391 31325 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0c9f82ffab643f9865b6ef08d2191fb [sys.catalog]: SysCatalogTable state changed. Reason: New leader c0c9f82ffab643f9865b6ef08d2191fb. Latest consensus state: current_term: 1 leader_uuid: "c0c9f82ffab643f9865b6ef08d2191fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0c9f82ffab643f9865b6ef08d2191fb" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 41793 } } peers { permanent_uuid: "46b6aa835c274b3fbdca0b6d2ff09ac7" member_type: VOTER last_known_addr { host: "127.27.161.61" port: 42455 } } peers { permanent_uuid: "97352ed50d564be5aeca141f1cb3218e" member_type: VOTER last_known_addr { host: "127.27.161.60" port: 35303 } } }
I20260810 03:32:03.118999 31325 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0c9f82ffab643f9865b6ef08d2191fb [sys.catalog]: This master's current role is: LEADER
I20260810 03:32:03.131860 31331 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 03:32:03.134346 31331 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 03:32:03.149861 31242 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 97352ed50d564be5aeca141f1cb3218e [term 1 FOLLOWER]: Refusing update from remote peer c0c9f82ffab643f9865b6ef08d2191fb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:32:03.151435 31325 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c0c9f82ffab643f9865b6ef08d2191fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "97352ed50d564be5aeca141f1cb3218e" member_type: VOTER last_known_addr { host: "127.27.161.60" port: 35303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:32:03.153982 31166 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.161.60" port: 35303 } attempt: 5
Aug 10 03:32:03 dist-test-slave-vpvm krb5kdc[30932](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332722, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.161.61@KRBTEST.COM for kudu/127.27.161.60@KRBTEST.COM
I20260810 03:32:03.173593 31169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:32:03.154996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.161.60:35303 (local address 127.0.0.1:47548)
0810 03:32:03.155432 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0810 03:32:03.155454 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0810 03:32:03.155578 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:32:03.156483 (+ 905us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:32:03.156487 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:32:03.157016 (+ 529us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:32:03.157247 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:32:03.157262 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:32:03.158787 (+ 1525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:32:03.158793 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:32:03.160266 (+ 1473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:32:03.160285 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:32:03.161505 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:32:03.161511 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:32:03.161788 (+ 277us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:32:03.162830 (+ 1042us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:32:03.162858 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:32:03.165924 (+ 3066us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:32:03.170324 (+ 4400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:32:03.170330 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:32:03.170333 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:32:03.170946 (+ 613us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:32:03.171438 (+ 492us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:32:03.171442 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:32:03.171445 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:32:03.171657 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:32:03.172433 (+ 776us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:32:03.172438 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:32:03.172647 (+ 209us) client_negotiation.cc:769] Sending connection context
0810 03:32:03.172818 (+ 171us) client_negotiation.cc:240] Negotiation successful
0810 03:32:03.173148 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294}
I20260810 03:32:03.174674 31282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:32:03.155798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47548 (local address 127.27.161.60:35303)
0810 03:32:03.155994 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0810 03:32:03.156000 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:32:03.156022 (+ 22us) server_negotiation.cc:407] Connection header received
0810 03:32:03.156082 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:32:03.156086 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:32:03.156171 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:32:03.156280 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:32:03.157446 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:32:03.158619 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:32:03.160459 (+ 1840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:32:03.161145 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:32:03.161273 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:32:03.166220 (+ 4947us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:32:03.166228 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:32:03.166233 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:32:03.166283 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:32:03.170101 (+ 3818us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:32:03.171110 (+ 1009us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:32:03.171115 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:32:03.171118 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:32:03.171286 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:32:03.171798 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:32:03.171802 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:32:03.171805 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:32:03.172211 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:32:03.172364 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0810 03:32:03.174048 (+ 1684us) server_negotiation.cc:299] Negotiation successful
0810 03:32:03.174305 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260810 03:32:03.174747 31288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:32:03.121798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53726 (local address 127.27.161.62:41793)
0810 03:32:03.122048 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0810 03:32:03.122055 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:32:03.130588 (+ 8533us) server_negotiation.cc:407] Connection header received
0810 03:32:03.130877 (+ 289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:32:03.130884 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:32:03.130975 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:32:03.131046 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:32:03.134689 (+ 3643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:32:03.136156 (+ 1467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:32:03.146604 (+ 10448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:32:03.147440 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:32:03.147562 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:32:03.161817 (+ 14255us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:32:03.161826 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:32:03.161831 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:32:03.161878 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:32:03.167101 (+ 5223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:32:03.168925 (+ 1824us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:32:03.168933 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:32:03.168937 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:32:03.169322 (+ 385us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:32:03.170438 (+ 1116us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:32:03.170477 (+ 39us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:32:03.170481 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:32:03.171373 (+ 892us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:32:03.172062 (+ 689us) server_negotiation.cc:1035] Waiting for connection context
0810 03:32:03.173429 (+ 1367us) server_negotiation.cc:299] Negotiation successful
0810 03:32:03.174004 (+ 575us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260810 03:32:03.184418 31166 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 46b6aa835c274b3fbdca0b6d2ff09ac7: Bootstrap starting.
I20260810 03:32:03.189427 31166 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 46b6aa835c274b3fbdca0b6d2ff09ac7: Neither blocks nor log segments found. Creating new log.
I20260810 03:32:03.194937 31166 log.cc:824] T 00000000000000000000000000000000 P 46b6aa835c274b3fbdca0b6d2ff09ac7: Log is configured to *not* fsync() on all Append() calls
I20260810 03:32:03.200943 31291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:32:03.121575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50376 (local address 127.27.161.61:42455)
0810 03:32:03.121779 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0810 03:32:03.121786 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:32:03.129062 (+ 7276us) server_negotiation.cc:407] Connection header received
0810 03:32:03.129583 (+ 521us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:32:03.129591 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:32:03.129695 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:32:03.129782 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:32:03.143326 (+ 13544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:32:03.144509 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:32:03.158535 (+ 14026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:32:03.159327 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:32:03.159467 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:32:03.172217 (+ 12750us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:32:03.172225 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:32:03.172231 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:32:03.172277 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:32:03.191741 (+ 19464us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:32:03.196533 (+ 4792us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:32:03.196541 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:32:03.196546 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:32:03.196915 (+ 369us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:32:03.197893 (+ 978us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:32:03.197899 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:32:03.197903 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:32:03.198345 (+ 442us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:32:03.198600 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0810 03:32:03.199790 (+ 1190us) server_negotiation.cc:299] Negotiation successful
0810 03:32:03.200263 (+ 473us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260810 03:32:03.210528 31166 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 46b6aa835c274b3fbdca0b6d2ff09ac7: No bootstrap required, opened a new log
W20260810 03:32:03.213514 31212 tablet.cc:2369] T 00000000000000000000000000000000 P 97352ed50d564be5aeca141f1cb3218e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 03:32:03.213644 31317 sys_catalog.cc:455] T 00000000000000000000000000000000 P 97352ed50d564be5aeca141f1cb3218e [sys.catalog]: SysCatalogTable state changed. Reason: New leader c0c9f82ffab643f9865b6ef08d2191fb. Latest consensus state: current_term: 1 leader_uuid: "c0c9f82ffab643f9865b6ef08d2191fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0c9f82ffab643f9865b6ef08d2191fb" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 41793 } } peers { permanent_uuid: "46b6aa835c274b3fbdca0b6d2ff09ac7" member_type: VOTER last_known_addr { host: "127.27.161.61" port: 42455 } } peers { permanent_uuid: "97352ed50d564be5aeca141f1cb3218e" member_type: VOTER last_known_addr { host: "127.27.161.60" port: 35303 } } }
I20260810 03:32:03.214041 31317 sys_catalog.cc:458] T 00000000000000000000000000000000 P 97352ed50d564be5aeca141f1cb3218e [sys.catalog]: This master's current role is: FOLLOWER
I20260810 03:32:03.223295 31281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:32:03.128553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47544 (local address 127.27.161.60:35303)
0810 03:32:03.128749 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0810 03:32:03.128755 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:32:03.148092 (+ 19337us) server_negotiation.cc:407] Connection header received
0810 03:32:03.152277 (+ 4185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:32:03.152284 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:32:03.152412 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:32:03.152519 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:32:03.167024 (+ 14505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:32:03.168362 (+ 1338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:32:03.201098 (+ 32736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:32:03.201910 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:32:03.202482 (+ 572us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:32:03.208121 (+ 5639us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:32:03.208130 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:32:03.208135 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:32:03.208183 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:32:03.215586 (+ 7403us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:32:03.217698 (+ 2112us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:32:03.217707 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:32:03.217711 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:32:03.218125 (+ 414us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:32:03.219108 (+ 983us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:32:03.219117 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:32:03.219121 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:32:03.220000 (+ 879us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:32:03.220614 (+ 614us) server_negotiation.cc:1035] Waiting for connection context
0810 03:32:03.221828 (+ 1214us) server_negotiation.cc:299] Negotiation successful
0810 03:32:03.222408 (+ 580us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260810 03:32:03.226110 31340 mvcc.cc:205] Tried to move back new op lower bound from 7316818834006810624 to 7316818833881640960. Current Snapshot: MvccSnapshot[applied={T|T < 7316818834006810624}]
I20260810 03:32:03.229858 31317 sys_catalog.cc:455] T 00000000000000000000000000000000 P 97352ed50d564be5aeca141f1cb3218e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c0c9f82ffab643f9865b6ef08d2191fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0c9f82ffab643f9865b6ef08d2191fb" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 41793 } } peers { permanent_uuid: "46b6aa835c274b3fbdca0b6d2ff09ac7" member_type: VOTER last_known_addr { host: "127.27.161.61" port: 42455 } } peers { permanent_uuid: "97352ed50d564be5aeca141f1cb3218e" member_type: VOTER last_known_addr { host: "127.27.161.60" port: 35303 } } }
I20260810 03:32:03.230203 31317 sys_catalog.cc:458] T 00000000000000000000000000000000 P 97352ed50d564be5aeca141f1cb3218e [sys.catalog]: This master's current role is: FOLLOWER
I20260810 03:32:03.236795 31346 mvcc.cc:205] Tried to move back new op lower bound from 7316818834006810624 to 7316818833881640960. Current Snapshot: MvccSnapshot[applied={T|T < 7316818834006810624 or (T in {7316818834006810624})}]
I20260810 03:32:03.245187 31292 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0c9f82ffab643f9865b6ef08d2191fb [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c0c9f82ffab643f9865b6ef08d2191fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0c9f82ffab643f9865b6ef08d2191fb" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 41793 } } peers { permanent_uuid: "46b6aa835c274b3fbdca0b6d2ff09ac7" member_type: VOTER last_known_addr { host: "127.27.161.61" port: 42455 } } peers { permanent_uuid: "97352ed50d564be5aeca141f1cb3218e" member_type: VOTER last_known_addr { host: "127.27.161.60" port: 35303 } } }
I20260810 03:32:03.245646 31292 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0c9f82ffab643f9865b6ef08d2191fb [sys.catalog]: This master's current role is: LEADER
I20260810 03:32:03.247328 31331 catalog_manager.cc:1347] Generated new cluster ID: f159cf1be80a4d14a2f24da7d09bda13
I20260810 03:32:03.247442 31331 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 03:32:03.256747 31166 raft_consensus.cc:348] T 00000000000000000000000000000000 P 46b6aa835c274b3fbdca0b6d2ff09ac7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0c9f82ffab643f9865b6ef08d2191fb" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 41793 } } peers { permanent_uuid: "46b6aa835c274b3fbdca0b6d2ff09ac7" member_type: VOTER last_known_addr { host: "127.27.161.61" port: 42455 } } peers { permanent_uuid: "97352ed50d564be5aeca141f1cb3218e" member_type: VOTER last_known_addr { host: "127.27.161.60" port: 35303 } }
I20260810 03:32:03.257429 31166 raft_consensus.cc:729] T 00000000000000000000000000000000 P 46b6aa835c274b3fbdca0b6d2ff09ac7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46b6aa835c274b3fbdca0b6d2ff09ac7, State: Initialized, Role: FOLLOWER
I20260810 03:32:03.258113 31166 consensus_queue.cc:260] T 00000000000000000000000000000000 P 46b6aa835c274b3fbdca0b6d2ff09ac7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0c9f82ffab643f9865b6ef08d2191fb" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 41793 } } peers { permanent_uuid: "46b6aa835c274b3fbdca0b6d2ff09ac7" member_type: VOTER last_known_addr { host: "127.27.161.61" port: 42455 } } peers { permanent_uuid: "97352ed50d564be5aeca141f1cb3218e" member_type: VOTER last_known_addr { host: "127.27.161.60" port: 35303 } }
I20260810 03:32:03.265499 31355 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46b6aa835c274b3fbdca0b6d2ff09ac7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0c9f82ffab643f9865b6ef08d2191fb" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 41793 } } peers { permanent_uuid: "46b6aa835c274b3fbdca0b6d2ff09ac7" member_type: VOTER last_known_addr { host: "127.27.161.61" port: 42455 } } peers { permanent_uuid: "97352ed50d564be5aeca141f1cb3218e" member_type: VOTER last_known_addr { host: "127.27.161.60" port: 35303 } } }
I20260810 03:32:03.268293 31355 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46b6aa835c274b3fbdca0b6d2ff09ac7 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 03:32:03.268187 31133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c0c9f82ffab643f9865b6ef08d2191fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "46b6aa835c274b3fbdca0b6d2ff09ac7"
I20260810 03:32:03.268103 31166 sys_catalog.cc:564] T 00000000000000000000000000000000 P 46b6aa835c274b3fbdca0b6d2ff09ac7 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:32:03.268757 31133 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 46b6aa835c274b3fbdca0b6d2ff09ac7 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:32:03.272429 31133 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 46b6aa835c274b3fbdca0b6d2ff09ac7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0c9f82ffab643f9865b6ef08d2191fb in term 1.
I20260810 03:32:03.273439 31132 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 46b6aa835c274b3fbdca0b6d2ff09ac7 [term 1 FOLLOWER]: Refusing update from remote peer c0c9f82ffab643f9865b6ef08d2191fb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:32:03.275158 31325 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c0c9f82ffab643f9865b6ef08d2191fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "46b6aa835c274b3fbdca0b6d2ff09ac7" member_type: VOTER last_known_addr { host: "127.27.161.61" port: 42455 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:32:03.279765 31359 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260810 03:32:03.288821 31331 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 03:32:03.290822 31331 catalog_manager.cc:1504] Loading token signing keys...
I20260810 03:32:03.287242 31178 master_runner.cc:186] Error getting master registration for 127.27.161.61:42455: OK, instance_id { permanent_uuid: "46b6aa835c274b3fbdca0b6d2ff09ac7" instance_seqno: 1786332722415020 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260810 03:32:03.295347 31178 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 03:32:03.310684 31331 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c0c9f82ffab643f9865b6ef08d2191fb: Generated new TSK 0
I20260810 03:32:03.311911 31331 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 03:32:03.319346 31355 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46b6aa835c274b3fbdca0b6d2ff09ac7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c0c9f82ffab643f9865b6ef08d2191fb. Latest consensus state: current_term: 1 leader_uuid: "c0c9f82ffab643f9865b6ef08d2191fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0c9f82ffab643f9865b6ef08d2191fb" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 41793 } } peers { permanent_uuid: "46b6aa835c274b3fbdca0b6d2ff09ac7" member_type: VOTER last_known_addr { host: "127.27.161.61" port: 42455 } } peers { permanent_uuid: "97352ed50d564be5aeca141f1cb3218e" member_type: VOTER last_known_addr { host: "127.27.161.60" port: 35303 } } }
I20260810 03:32:03.319679 31355 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46b6aa835c274b3fbdca0b6d2ff09ac7 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 03:32:03.319578 31292 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0c9f82ffab643f9865b6ef08d2191fb [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c0c9f82ffab643f9865b6ef08d2191fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0c9f82ffab643f9865b6ef08d2191fb" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 41793 } } peers { permanent_uuid: "46b6aa835c274b3fbdca0b6d2ff09ac7" member_type: VOTER last_known_addr { host: "127.27.161.61" port: 42455 } } peers { permanent_uuid: "97352ed50d564be5aeca141f1cb3218e" member_type: VOTER last_known_addr { host: "127.27.161.60" port: 35303 } } }
I20260810 03:32:03.319885 31292 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0c9f82ffab643f9865b6ef08d2191fb [sys.catalog]: This master's current role is: LEADER
I20260810 03:32:03.320119 31367 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46b6aa835c274b3fbdca0b6d2ff09ac7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c0c9f82ffab643f9865b6ef08d2191fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0c9f82ffab643f9865b6ef08d2191fb" member_type: VOTER last_known_addr { host: "127.27.161.62" port: 41793 } } peers { permanent_uuid: "46b6aa835c274b3fbdca0b6d2ff09ac7" member_type: VOTER last_known_addr { host: "127.27.161.61" port: 42455 } } peers { permanent_uuid: "97352ed50d564be5aeca141f1cb3218e" member_type: VOTER last_known_addr { host: "127.27.161.60" port: 35303 } } }
I20260810 03:32:03.320587 31367 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46b6aa835c274b3fbdca0b6d2ff09ac7 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 03:32:03.362260 31386 catalog_manager.cc:1259] Loaded cluster ID: f159cf1be80a4d14a2f24da7d09bda13
I20260810 03:32:03.362498 31386 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 46b6aa835c274b3fbdca0b6d2ff09ac7: loading cluster ID for follower catalog manager: success
I20260810 03:32:03.366712 31386 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 46b6aa835c274b3fbdca0b6d2ff09ac7: acquiring CA information for follower catalog manager: success
I20260810 03:32:03.369729 31386 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 46b6aa835c274b3fbdca0b6d2ff09ac7: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260810 03:32:03.459558 31288 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:32:03.454052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53734 (local address 127.27.161.62:41793)
0810 03:32:03.454242 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0810 03:32:03.454248 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:32:03.457610 (+ 3362us) server_negotiation.cc:407] Connection header received
0810 03:32:03.457850 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:32:03.457856 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:32:03.458017 (+ 161us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:32:03.458146 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:32:03.459271 (+ 1125us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53734: BlockingRecv error: recv got EOF from 127.0.0.1:53734 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260810 03:32:03.461305 31281 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:32:03.458065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47564 (local address 127.27.161.60:35303)
0810 03:32:03.458465 (+ 400us) server_negotiation.cc:206] Beginning negotiation
0810 03:32:03.458472 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:32:03.459764 (+ 1292us) server_negotiation.cc:407] Connection header received
0810 03:32:03.459840 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:32:03.459845 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:32:03.459927 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:32:03.460024 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:32:03.461057 (+ 1033us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47564: BlockingRecv error: recv got EOF from 127.0.0.1:47564 (error 108)
Metrics: {"server-negotiator.queue_time_us":248}
W20260810 03:32:03.463125 31291 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:32:03.453296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50390 (local address 127.27.161.61:42455)
0810 03:32:03.453509 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0810 03:32:03.453515 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:32:03.457313 (+ 3798us) server_negotiation.cc:407] Connection header received
0810 03:32:03.459614 (+ 2301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:32:03.459621 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:32:03.459733 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:32:03.459809 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:32:03.462848 (+ 3039us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50390: BlockingRecv error: recv got EOF from 127.0.0.1:50390 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
I20260810 03:32:03.482324 31291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:32:03.470316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50396 (local address 127.27.161.61:42455)
0810 03:32:03.470503 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0810 03:32:03.470509 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:32:03.474536 (+ 4027us) server_negotiation.cc:407] Connection header received
0810 03:32:03.474794 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:32:03.474800 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:32:03.475091 (+ 291us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:32:03.475299 (+ 208us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 03:32:03.475807 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:32:03.476980 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:32:03.479154 (+ 2174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:32:03.479865 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:32:03.480017 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:32:03.480659 (+ 642us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 03:32:03.481505 (+ 846us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 03:32:03.481671 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0810 03:32:03.481794 (+ 123us) server_negotiation.cc:299] Negotiation successful
0810 03:32:03.481907 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
W20260810 03:32:03.492029 31281 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:32:03.489450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47572 (local address 127.27.161.60:35303)
0810 03:32:03.489652 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0810 03:32:03.489660 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 03:32:03.490235 (+ 575us) server_negotiation.cc:407] Connection header received
0810 03:32:03.490613 (+ 378us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:32:03.490619 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:32:03.490756 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:32:03.490862 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:32:03.491764 (+ 902us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47572: BlockingRecv error: recv got EOF from 127.0.0.1:47572 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
I20260810 03:32:03.503509 31291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:32:03.496548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50402 (local address 127.27.161.61:42455)
0810 03:32:03.496736 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0810 03:32:03.496742 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:32:03.497268 (+ 526us) server_negotiation.cc:407] Connection header received
0810 03:32:03.497458 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:32:03.497464 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:32:03.497561 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:32:03.497664 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 03:32:03.498158 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:32:03.499300 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:32:03.501283 (+ 1983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:32:03.502028 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:32:03.502186 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:32:03.502667 (+ 481us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 03:32:03.502814 (+ 147us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 03:32:03.502903 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0810 03:32:03.503071 (+ 168us) server_negotiation.cc:299] Negotiation successful
0810 03:32:03.503164 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
W20260810 03:32:03.513521 31281 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:32:03.510970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47586 (local address 127.27.161.60:35303)
0810 03:32:03.511187 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0810 03:32:03.511194 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:32:03.511708 (+ 514us) server_negotiation.cc:407] Connection header received
0810 03:32:03.511910 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:32:03.511915 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:32:03.512030 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:32:03.512142 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:32:03.513222 (+ 1080us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47586: BlockingRecv error: recv got EOF from 127.0.0.1:47586 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
I20260810 03:32:03.531459 31291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:32:03.524311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50404 (local address 127.27.161.61:42455)
0810 03:32:03.524544 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0810 03:32:03.524551 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:32:03.525138 (+ 587us) server_negotiation.cc:407] Connection header received
0810 03:32:03.525364 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:32:03.525370 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:32:03.525473 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:32:03.525576 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 03:32:03.526083 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:32:03.527214 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:32:03.529221 (+ 2007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:32:03.529890 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:32:03.530090 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:32:03.530590 (+ 500us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 03:32:03.530778 (+ 188us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 03:32:03.530871 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0810 03:32:03.531052 (+ 181us) server_negotiation.cc:299] Negotiation successful
0810 03:32:03.531143 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
W20260810 03:32:03.550413 31281 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:32:03.548083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47596 (local address 127.27.161.60:35303)
0810 03:32:03.548297 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0810 03:32:03.548304 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:32:03.548701 (+ 397us) server_negotiation.cc:407] Connection header received
0810 03:32:03.548928 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:32:03.548934 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:32:03.549039 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:32:03.549151 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:32:03.550115 (+ 964us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47596: BlockingRecv error: recv got EOF from 127.0.0.1:47596 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
I20260810 03:32:03.565866 31291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:32:03.558770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50408 (local address 127.27.161.61:42455)
0810 03:32:03.559017 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0810 03:32:03.559024 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:32:03.559581 (+ 557us) server_negotiation.cc:407] Connection header received
0810 03:32:03.559796 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:32:03.559802 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:32:03.559905 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:32:03.560013 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 03:32:03.560517 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:32:03.561745 (+ 1228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:32:03.563761 (+ 2016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:32:03.564441 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:32:03.564579 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:32:03.565090 (+ 511us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 03:32:03.565236 (+ 146us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 03:32:03.565314 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0810 03:32:03.565471 (+ 157us) server_negotiation.cc:299] Negotiation successful
0810 03:32:03.565563 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127}
W20260810 03:32:03.578203 31281 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:32:03.575879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47608 (local address 127.27.161.60:35303)
0810 03:32:03.576148 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0810 03:32:03.576155 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:32:03.576683 (+ 528us) server_negotiation.cc:407] Connection header received
0810 03:32:03.576866 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:32:03.576872 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:32:03.576973 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:32:03.577096 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:32:03.577945 (+ 849us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47608: BlockingRecv error: recv got EOF from 127.0.0.1:47608 (error 108)
Metrics: {"server-negotiator.queue_time_us":146}
I20260810 03:32:03.598855 31291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:32:03.591685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50416 (local address 127.27.161.61:42455)
0810 03:32:03.591886 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0810 03:32:03.591893 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:32:03.592466 (+ 573us) server_negotiation.cc:407] Connection header received
0810 03:32:03.592709 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:32:03.592716 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:32:03.592837 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:32:03.592957 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 03:32:03.593473 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:32:03.594664 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:32:03.596717 (+ 2053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:32:03.597416 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:32:03.597537 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:32:03.598055 (+ 518us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 03:32:03.598190 (+ 135us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 03:32:03.598271 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0810 03:32:03.598426 (+ 155us) server_negotiation.cc:299] Negotiation successful
0810 03:32:03.598523 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
W20260810 03:32:03.615572 31281 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:32:03.613482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47618 (local address 127.27.161.60:35303)
0810 03:32:03.613663 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0810 03:32:03.613670 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:32:03.614076 (+ 406us) server_negotiation.cc:407] Connection header received
0810 03:32:03.614293 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:32:03.614299 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:32:03.614394 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:32:03.614483 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:32:03.615321 (+ 838us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47618: BlockingRecv error: recv got EOF from 127.0.0.1:47618 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
I20260810 03:32:03.641371 31291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:32:03.635289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50422 (local address 127.27.161.61:42455)
0810 03:32:03.635503 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0810 03:32:03.635510 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:32:03.635830 (+ 320us) server_negotiation.cc:407] Connection header received
0810 03:32:03.636024 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:32:03.636030 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:32:03.636120 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:32:03.636239 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 03:32:03.636616 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:32:03.637795 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:32:03.639256 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:32:03.639951 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:32:03.640104 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:32:03.640576 (+ 472us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 03:32:03.640741 (+ 165us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 03:32:03.640838 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0810 03:32:03.640991 (+ 153us) server_negotiation.cc:299] Negotiation successful
0810 03:32:03.641077 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
W20260810 03:32:03.649865 31281 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:32:03.647738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47626 (local address 127.27.161.60:35303)
0810 03:32:03.647980 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0810 03:32:03.647987 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:32:03.648400 (+ 413us) server_negotiation.cc:407] Connection header received
0810 03:32:03.648626 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:32:03.648631 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:32:03.648711 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:32:03.648795 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:32:03.649624 (+ 829us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47626: BlockingRecv error: recv got EOF from 127.0.0.1:47626 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
I20260810 03:32:03.691825 31291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:32:03.685829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50426 (local address 127.27.161.61:42455)
0810 03:32:03.686072 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0810 03:32:03.686079 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:32:03.686373 (+ 294us) server_negotiation.cc:407] Connection header received
0810 03:32:03.686537 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:32:03.686546 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:32:03.686644 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:32:03.686774 (+ 130us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 03:32:03.687181 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:32:03.688366 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:32:03.689811 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:32:03.690496 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:32:03.690642 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:32:03.691014 (+ 372us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 03:32:03.691187 (+ 173us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 03:32:03.691287 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0810 03:32:03.691382 (+ 95us) server_negotiation.cc:299] Negotiation successful
0810 03:32:03.691474 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
W20260810 03:32:03.700215 31281 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:32:03.697968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47636 (local address 127.27.161.60:35303)
0810 03:32:03.698182 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0810 03:32:03.698188 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:32:03.698612 (+ 424us) server_negotiation.cc:407] Connection header received
0810 03:32:03.698782 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:32:03.698787 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:32:03.698871 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:32:03.698964 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:32:03.699955 (+ 991us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47636: BlockingRecv error: recv got EOF from 127.0.0.1:47636 (error 108)
Metrics: {"server-negotiator.queue_time_us":91}
I20260810 03:32:03.775305 31291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:32:03.768752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50440 (local address 127.27.161.61:42455)
0810 03:32:03.768999 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0810 03:32:03.769006 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:32:03.769390 (+ 384us) server_negotiation.cc:407] Connection header received
0810 03:32:03.769938 (+ 548us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:32:03.769944 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:32:03.770069 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:32:03.770200 (+ 131us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 03:32:03.770663 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:32:03.771739 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:32:03.773169 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:32:03.773872 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:32:03.774034 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:32:03.774446 (+ 412us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 03:32:03.774612 (+ 166us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 03:32:03.774696 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0810 03:32:03.774854 (+ 158us) server_negotiation.cc:299] Negotiation successful
0810 03:32:03.774942 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
W20260810 03:32:03.783432 31281 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:32:03.781406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47650 (local address 127.27.161.60:35303)
0810 03:32:03.781626 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0810 03:32:03.781633 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:32:03.781936 (+ 303us) server_negotiation.cc:407] Connection header received
0810 03:32:03.782162 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:32:03.782167 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:32:03.782247 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:32:03.782334 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:32:03.783182 (+ 848us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47650: BlockingRecv error: recv got EOF from 127.0.0.1:47650 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
I20260810 03:32:03.920264 31291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:32:03.915184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50444 (local address 127.27.161.61:42455)
0810 03:32:03.915416 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0810 03:32:03.915422 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:32:03.915704 (+ 282us) server_negotiation.cc:407] Connection header received
0810 03:32:03.915856 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:32:03.915861 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:32:03.915945 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:32:03.916074 (+ 129us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 03:32:03.916614 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:32:03.917312 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:32:03.918726 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:32:03.919173 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:32:03.919279 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:32:03.919638 (+ 359us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 03:32:03.919746 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 03:32:03.919815 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0810 03:32:03.919908 (+ 93us) server_negotiation.cc:299] Negotiation successful
0810 03:32:03.919981 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
W20260810 03:32:03.928244 31281 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:32:03.926329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47664 (local address 127.27.161.60:35303)
0810 03:32:03.926545 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0810 03:32:03.926553 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 03:32:03.926860 (+ 307us) server_negotiation.cc:407] Connection header received
0810 03:32:03.927062 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:32:03.927068 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:32:03.927145 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:32:03.927234 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:32:03.928059 (+ 825us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47664: BlockingRecv error: recv got EOF from 127.0.0.1:47664 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
I20260810 03:32:04.196440 31291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:32:04.191625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50450 (local address 127.27.161.61:42455)
0810 03:32:04.191807 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0810 03:32:04.191811 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:32:04.192044 (+ 233us) server_negotiation.cc:407] Connection header received
0810 03:32:04.192217 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:32:04.192221 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:32:04.192291 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:32:04.192380 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 03:32:04.192735 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:32:04.193472 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:32:04.194849 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:32:04.195252 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:32:04.195359 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:32:04.195787 (+ 428us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 03:32:04.195921 (+ 134us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 03:32:04.196021 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0810 03:32:04.196097 (+ 76us) server_negotiation.cc:299] Negotiation successful
0810 03:32:04.196173 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
W20260810 03:32:04.204926 31281 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:32:04.202794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47672 (local address 127.27.161.60:35303)
0810 03:32:04.203061 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0810 03:32:04.203070 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 03:32:04.203320 (+ 250us) server_negotiation.cc:407] Connection header received
0810 03:32:04.203556 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:32:04.203563 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:32:04.203648 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:32:04.203749 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:32:04.204659 (+ 910us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47672: BlockingRecv error: recv got EOF from 127.0.0.1:47672 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
I20260810 03:32:04.284255 31359 catalog_manager.cc:1259] Loaded cluster ID: f159cf1be80a4d14a2f24da7d09bda13
I20260810 03:32:04.284389 31359 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 97352ed50d564be5aeca141f1cb3218e: loading cluster ID for follower catalog manager: success
I20260810 03:32:04.287161 31359 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 97352ed50d564be5aeca141f1cb3218e: acquiring CA information for follower catalog manager: success
I20260810 03:32:04.289081 31359 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 97352ed50d564be5aeca141f1cb3218e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 03:32:04.725447 31573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:32:04.720446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50466 (local address 127.27.161.61:42455)
0810 03:32:04.720940 (+ 494us) server_negotiation.cc:206] Beginning negotiation
0810 03:32:04.720945 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:32:04.720967 (+ 22us) server_negotiation.cc:407] Connection header received
0810 03:32:04.721166 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:32:04.721170 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:32:04.721243 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:32:04.721331 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 03:32:04.721776 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:32:04.722566 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:32:04.723937 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:32:04.724344 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:32:04.724447 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:32:04.724779 (+ 332us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 03:32:04.724901 (+ 122us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 03:32:04.724970 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0810 03:32:04.725060 (+ 90us) server_negotiation.cc:299] Negotiation successful
0810 03:32:04.725130 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":390,"thread_start_us":276,"threads_started":1}
I20260810 03:32:04.736800 31583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:32:04.730994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47682 (local address 127.27.161.60:35303)
0810 03:32:04.731454 (+ 460us) server_negotiation.cc:206] Beginning negotiation
0810 03:32:04.731459 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:32:04.731528 (+ 69us) server_negotiation.cc:407] Connection header received
0810 03:32:04.731696 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:32:04.731700 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:32:04.731869 (+ 169us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:32:04.731977 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 03:32:04.732479 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:32:04.733223 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:32:04.734759 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:32:04.735186 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:32:04.735285 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:32:04.735740 (+ 455us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 03:32:04.736267 (+ 527us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 03:32:04.736380 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0810 03:32:04.736485 (+ 105us) server_negotiation.cc:299] Negotiation successful
0810 03:32:04.736558 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":356,"thread_start_us":239,"threads_started":1}
I20260810 03:32:04.749900 31573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:32:04.742954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50470 (local address 127.27.161.61:42455)
0810 03:32:04.743271 (+ 317us) server_negotiation.cc:206] Beginning negotiation
0810 03:32:04.743279 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 03:32:04.744610 (+ 1331us) server_negotiation.cc:407] Connection header received
0810 03:32:04.744796 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:32:04.744802 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:32:04.744911 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:32:04.744989 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 03:32:04.745548 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:32:04.746523 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:32:04.748251 (+ 1728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:32:04.748669 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:32:04.748775 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:32:04.749189 (+ 414us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 03:32:04.749294 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 03:32:04.749362 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0810 03:32:04.749507 (+ 145us) server_negotiation.cc:299] Negotiation successful
0810 03:32:04.749599 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130}
I20260810 03:32:04.753146 31583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:32:04.743652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47690 (local address 127.27.161.60:35303)
0810 03:32:04.743824 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0810 03:32:04.743829 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:32:04.746531 (+ 2702us) server_negotiation.cc:407] Connection header received
0810 03:32:04.746702 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:32:04.746708 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:32:04.746812 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:32:04.746918 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 03:32:04.747556 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:32:04.748725 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:32:04.751027 (+ 2302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:32:04.751600 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:32:04.751681 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:32:04.752355 (+ 674us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 03:32:04.752460 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 03:32:04.752546 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0810 03:32:04.752765 (+ 219us) server_negotiation.cc:299] Negotiation successful
0810 03:32:04.752849 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260810 03:32:04.756505 31594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:32:04.742891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53750 (local address 127.27.161.62:41793)
0810 03:32:04.750488 (+ 7597us) server_negotiation.cc:206] Beginning negotiation
0810 03:32:04.750494 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:32:04.750514 (+ 20us) server_negotiation.cc:407] Connection header received
0810 03:32:04.750629 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:32:04.750635 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:32:04.750862 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:32:04.751627 (+ 765us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 03:32:04.751776 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:32:04.752674 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:32:04.754473 (+ 1799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:32:04.754896 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:32:04.754995 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:32:04.755523 (+ 528us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 03:32:04.755895 (+ 372us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 03:32:04.755988 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0810 03:32:04.756136 (+ 148us) server_negotiation.cc:299] Negotiation successful
0810 03:32:04.756211 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7472,"thread_start_us":7353,"threads_started":1}
I20260810 03:32:04.761128 28292 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iBH5/build/asan/bin/kudu pid 30958
I20260810 03:32:04.762224 31047 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:32:04.874588 28292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iBH5/build/asan/bin/kudu with pid 30958
I20260810 03:32:04.908016 28292 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iBH5/build/asan/bin/kudu pid 31062
I20260810 03:32:04.909195 31161 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:32:05.023233 28292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iBH5/build/asan/bin/kudu with pid 31062
I20260810 03:32:05.048035 28292 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task24iBH5/build/asan/bin/kudu pid 31178
I20260810 03:32:05.049177 31272 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:32:05.158691 28292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task24iBH5/build/asan/bin/kudu with pid 31178
2026-08-10T03:32:05Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (5427 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (10310 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (388730 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST